Run Reports

[jwodder]

This issue is for listing information about successful runs of s3invsync so that general performance etc. can be tracked.

CC @yarikoptic @aaronkanzer

[jwodder]

• Commit: fa1362d (PR Ignore empty directory entries in CSVs #96)
• Host: typhon (Debian bookworm, 32 cores)
• Bucket: dandiarchive
• Manifest: 2024-12-19T01-00Z
• Path filter: ^zarr/aa (20 Zarrs, 2718289 files, 1819 GB)
• Start time: 2024-12-20 08:44:43 EDT
• First (non-CSV) file download started: 2024-12-20 09:02:38 EDT
• Last (non-CSV) file download finished: 2024-12-20 19:14:24 EDT
• End time: 2024-12-20 19:15:26 EDT
• Runtime: 10 hours, 30 minutes, 43.59 seconds
• Maximum memory usage: 254.6 MB
• Maximum CPU usage: 1858%
• Note: The code was patched to eliminate log messages about keys not matching --path-filter in order to reduce logging noise.

info.json file produced by duct

{
  "command": "target/release/s3invsync --path-filter ^zarr/aa -l TRACE s3://dandiarchive/dandiarchive/dandiarchive/ /mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/backup",
  "system": {
    "cpu_total": 32,
    "memory_total": 1081801367552,
    "hostname": "typhon",
    "uid": 1012,
    "user": "jwodder"
  },
  "env": {},
  "gpu": [
    {
      "index": "0",
      "name": "NVIDIA A100-PCIE-40GB",
      "bus_id": "00000000:31:00.0",
      "driver_version": "560.35.05",
      "memory.total": "40960 MiB",
      "compute_mode": "Default"
    }
  ],
  "duct_version": "0.8.0",
  "schema_version": "0.2.0",
  "execution_summary": {
    "exit_code": 0,
    "command": "target/release/s3invsync --path-filter ^zarr/aa -l TRACE s3://dandiarchive/dandiarchive/dandiarchive/ /mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/backup",
    "logs_prefix": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/",
    "wall_clock_time": 37843.58987116814,
    "peak_rss": 254562304,
    "average_rss": 246478258.0884677,
    "peak_vsz": 3841757184,
    "average_vsz": 3741354899.0543704,
    "peak_pmem": 0.0,
    "average_pmem": 0.0,
    "peak_pcpu": 1858.0,
    "average_pcpu": 403.07713231904233,
    "num_samples": 36873,
    "num_reports": 632,
    "start_time": 1734702283.3706026,
    "end_time": 1734740126.9604738
  },
  "output_paths": {
    "stdout": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/stdout",
    "stderr": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/stderr",
    "usage": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/usage.json",
    "info": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/info.json",
    "prefix": "/mnt/tmp/jwodder/2024.12.20.13.44.43Z-gh-95-fa1362d/duct/"
  }
}

[jwodder]

• Commit: e13ca49
• Host: typhon (Debian bookworm, 32 cores)
• Bucket: dandiarchive
• Manifest: 2024-12-20T01-00Z
• Path filter: ^zarr/aaa (2 Zarrs, 177149 files, 84 GB)
• Start time: 2024-12-20 21:54:03 EDT
• First (non-CSV) file download started: 2024-12-20 22:14:20 EDT
• Last (non-CSV) file download finished: 2024-12-20 23:19:38 EDT
• End time: 2024-12-20 23:20:02 EDT
• Runtime: 1 hour, 25 minutes, 59.334 seconds
• Maximum memory usage: 242.6 MB
• Maximum CPU usage: 1846.0%
• Note: The code was patched to eliminate log messages about keys not matching --path-filter in order to reduce logging noise.

info.json file produced by duct

{
  "command": "target/release/s3invsync --path-filter ^zarr/aaa -l TRACE s3://dandiarchive/dandiarchive/dandiarchive/ /mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/backup",
  "system": {
    "cpu_total": 32,
    "memory_total": 1081801367552,
    "hostname": "typhon",
    "uid": 1012,
    "user": "jwodder"
  },
  "env": {},
  "gpu": [
    {
      "index": "0",
      "name": "NVIDIA A100-PCIE-40GB",
      "bus_id": "00000000:31:00.0",
      "driver_version": "560.35.05",
      "memory.total": "40960 MiB",
      "compute_mode": "Default"
    }
  ],
  "duct_version": "0.8.0",
  "schema_version": "0.2.0",
  "execution_summary": {
    "exit_code": 0,
    "command": "target/release/s3invsync --path-filter ^zarr/aaa -l TRACE s3://dandiarchive/dandiarchive/dandiarchive/ /mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/backup",
    "logs_prefix": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/",
    "wall_clock_time": 5159.334126710892,
    "peak_rss": 242601984,
    "average_rss": 219742777.31875098,
    "peak_vsz": 3562713088,
    "average_vsz": 3561735612.5003014,
    "peak_pmem": 0.0,
    "average_pmem": 0.0,
    "peak_pcpu": 1846.0,
    "average_pcpu": 1131.5855040763593,
    "num_samples": 5029,
    "num_reports": 87,
    "start_time": 1734749643.3861423,
    "end_time": 1734754802.720269
  },
  "output_paths": {
    "stdout": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/stdout",
    "stderr": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/stderr",
    "usage": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/usage.json",
    "info": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/info.json",
    "prefix": "/mnt/tmp/jwodder/s3invsync/2024.12.21.02.54.03Z-main-e13ca49/duct/"
  }
}

[yarikoptic]

did you manually extracted those entries for "... (non-CSV) file download" or there might be a helper which given duct's info.json would extract all of that from that .json + output files containing logs?

[jwodder]

@yarikoptic I just grepped the logs with:

grep 'download_item:download_object' stderr | head -n1
grep 'download_item:download_object.*Finished' stderr | tail -n1