Loading Data from S3

[hilt86]

Thanks for making Zat!

I would like to be able to centrally store all my data in Amazon S3 and access then using Zat, however currently Zat expects an actual filesystem. Is it possible to use the native-ish support for S3 in Zat please?

[brifordwylie]

This is a super awesome suggestion.... putting this on the top of my open source queue 👍

[hilt86]

Thanks! At the moment I've modified the /opt/zeek/share/zeekctl/scripts/archive-log log to call a small python script that converts the file to a parquet file on Amazon S3. For Analysis I use Google Colab like so :

fs = s3fs.S3FileSystem(anon=False)
filz = fs.find('s3://parquet.my.org/2022/10/')

fileList = []
for f in filz:
  if "dns" in f:
    fileList.append('s3://' + f)

Then I use pc.concat() to combine all the minis into one big dataframe for analysis :

df = pd.concat((pd.read_parquet(f, engine='fastparquet') for f in fileList), ignore_index=False)