We are currently working on porting this changelog to the specifications in Keep a Changelog. This project adheres to Semantic Versioning.
CocoDataset.conform
now returns a reference toself
- Support segmentations in legacy
COCO.loadRes
- Perterb no longer fails if segmentations are not present
- Subset now correctly handles tracks
segmentation_metrics
now has the ability to dump components of its visualization for more flexible figure aggregation.segmentation_metrics
now has the ability to specify a subset of images to evaluate on.- conforming to the legacy coco format will now remove holes and warn.
kwcoco modify_categories
now uses kwutil and YAML arguments to specify keep / remove.- The return type of
_ensure_imgsize
changed from aList[Dict]
to aDict
with summary information. The old return value is accessible via the"bad_images"
key in the new summary dictionary. - The
kwcoco.CategoryTree.coerce
now will recognize coco category lists. CocoDataset.union
will now preserve category ids if possible.
- the
loadImgs
,loadAnns
, andloadCats
functions in the compat dataset now do the same thing as pycocotools
- Fix broken paths in camvid converter
- fixed issues when converting segmentations to legacy mscoco format
- Issue with
image_filepath
in CocoImage / CocoAsset
- Added
CocoDataset.add_categories
- Users can now specify sensorchan for toydata videos to get a custom sensor/channel arrangement
- Added
flatten
method toObjectGroups
. - Add public
rebuild_index
function which should be used instead of_build_index
.
- Set verbose default of reroot to 0 to match documentation.
rich
is now a runtime requirement.
- Changed
visual_stats
toplot_stats
. - Moved kw18 and kpf to a new
formats
submodule. - kwcoco show now respects video frame order if available
- modified scheme so supercategory is now allowed to be an empty string
- Added basic helpers for converting from labelme and VOC annotations
- Added finish-install helper script.
- New
load
classmethod (analogous todump
) as a placeholder for a potential future refactor. - Ported segmentation metrics from geowatch. Eventually metrics may be ported to a new package: kwmetrics in order to improve modularity and keep components well-scoped.
- Added
out_fpath
tococo_eval
CLI signature. - Added
video_ids
argument toCocoDataset.subset
- Added
.objs_iter()
method toObjectList1D
as an alternative to.objs
- Added
.get_iter(...)
method toObjectList1D
as an alternative to.get(...)
- Added
.coco_images_iter()
method toImages
as an alternative to.coco_images
kwcoco/cli/coco_visual_stats.py
CocoDataset.add_track
will now pick a name using a uuid if one is unspecified.- Added more metrics to outputs in coco eval
- demodata can now be populated with random timestamps
- dict proxy items can now use the pop and delitem methods
- scriptconfig CLI interfaces have been modernized.
- CIFAR grabdata now respects dpath
CocoDataset.ensure_video
properly checks for duplicate video names now.
- Handle deprecated
scipy.integrate.trapz
- CIFAR grabdata now converts train and test variants of the dataset and includes channel information.
- Drop 3.6 and 3.7 support
- Optimize CocoObject1D get calls with SQL backends
- CocoObject1D.lookup with default now "works"-ish with SQL. Still experimental.
- Added workaround for an issue where a postgresql cache of a coco dataset is unable to remember its original json filepath.
- Fixed issue with
CocoSqlDatabase.disconnect
where a session would be dereferenced, but not closed. - Worked around issue in postgresql backend where track-ids were not decoded properly in their string form.
- Add CLI for
find_unregistered_images
- Add
video_id
argument toCocoDataset.annots
- Can now specify render dictionary with
main_ext
andmain_channels
torandom_video_dset
to control demo data file format and the channel codes that are used.
-
Prevent resource crash:
find_json_unserializable
now checks for circular references. -
Coco union now handles tracks.
- The
disjoint_tracks
inCocoDataset.union
now works on a "track-name" level. Old behavior where track-ids did not need a corresponding "track" entry is still preserved but may be removed in the future.
- Added
imgs_with_annots
to extended stats. - Background param now works as an argument to
demo
for vidshapes
- Added missing
name_to_video
index in sql dataset
- Added
take
andsubgraph
tokwcoco.CategoryTree
.
-
kwcoco.CategoryTree
no longer modifies node order to be sorted by id. If this causes something to break, old behavior can be temporarily restored by settingRESPECT_INPUT_ORDER = 0
, but this option will be removed in the future. -
kwcoco.CategoryTree.from_coco
and otherCategoryTree
constructors now respects the order in which the categories were given if id is not present.
- Add
datetime
attribute toCocoImage
. - Add
ensure_video
method toCocoDataset
. - Add
ensure_track
method toCocoDataset
. - Add
sort_values
method toObjectList1D
(and hence vectorizedImages
andAnnots
objects). - Add
--inplace
tokwcoco conform
.
- kwcoco.Coerce with sqlview now works for .kwcoco.zip files.
clear_annotations
now also clears tracks.Images.annots
now returnsAnnotGroups
where the underlyingAnnots
objects use id lists instead of id sets.- Issue with new matplotlib 3.8 and
show_image
.
kwcoco info
can now query images by name.- Add
io_workers
to kwcoco stats
- Fixed issue where imdelay would sometimes not respect antialias and interpolation
-
Top level track support in the dictionary dataset with
add_track
,remove_tracks
, andindex.name_to_track
. -
Single-object convenience classes for the major tables.
-
Default data structure for Python versions >= 3.7 is now a regular dictionary instead of OrderedDict. The behavior in 3.6 is unchanged.
-
Working towards a better and consistent CocoImage API without breaking compatibility. Added
iter_assets
and addedas_dict
kwargs toprimary_asset
. The goal is that these high level functions will return objects instead of dictionaries.
CocoImage.add_asset
can now accept an image_id property.
- Modified
kwcoco info
script to be able to stream the first n items in each table in a json compliant way.
- New CLI script:
coco_move_assets
. - New CLI script:
coco_info
.
- Issue in
coco_subset
CLI when copy-assets is on and the input is absolute paths. - reroot can now accept pathlib.Path arguments.
- Improved reroot logic.
- fixed issue in coco union where output paths were not rerooted correctly in some cases.
kwcoco validate
now reports the path to the first missing / corrupted image in the error.- Started work on new top-level track table.
- Started laying groundwork for moving assets to their own table.
- Moved examples out of the module and into the main repo.
- sliced getitems into a CocoObject1d (e.g. Images, Annotations, etc...) now returns a new CocoObject1d representing the subset instead of a list of integer ids.
- reroot now ignores leading "./" when looking for
old_prefix
CocoDataset.corrupted_images
now defaultscheck_aux
to TrueCocoDataset.corrupted_images
now can execute in parallel.CocoDataset.corrupted_images
now checks if it can read the entire image again.- Changed the folder "aux" to "auxiliary" in test data, due to windows issues.
CocoDataset.get_image_fpath
now returns a ubelt.Path objectCocoDataset.get_auxiliary_fpath
now returns a ubelt.Path objectCocoImage.iter_image_filepaths
now yields ubelt.Path objectsCocoImage.primary_image_filepath
now returns a ubelt.Path object
- Add id property to CocoObject1d
- Windows path fixes
- safer no longer tries to use temp files on windows
- Add
remember_parent
option to CocoDataset.union - Add
CocoDataset.coerce_multiple
andload_multiple
loader with parallel processing. - Add parallel IO to
kwcoco union
- Add util parallel and eval
- Issue in detection metrics when classes was not specified. Now uses the true detections to infer them if possible.
- Detection metric assign workers was never used
- Subsets of metrics now work when TN is not available.
- CLI backend changed to use scriptconfig ModalCLI
- CocoImage and CocoAsset use the new AliasDictProxy interface to smoothly transition between name conventions.
- The "assets" key is now an alias of "auxiliary" for CocoImage and we will switch to "assets" as the primary in the future.
- Toydata now generates RGB bands independently
- Change
__unstructured__
to_unstructured
for sqlalchemy 2.x compat
- demo data now has the ability to render on top of a background image.
- Fixed sqlalchemy 2.x compatibility
- The track index now works properly with SQL variants and returns ordreing
- Toydata now has better polygon generation for small image sizes, but is still not perfect.
- Changed the internal name used in compressed zip files.
Bug in CocoImage.add_asset
where json serialization was broken
- In
metrics.drawing
, make threshold colors more consistent - Added
category_names
property toAnnots1d
object. - Deprecate
CocoImage.add_auxiliary_item
in favor ofCocoImage.add_asset
- Added
annots
property to CocoImage - Added
add_annotation
toCocoImage
- Added
autofix
optionkwcoco reroot
- Better asset space support
- Querying
annots
by track-id now return them in a sorted order.
- Can now ask for the resolution of an asset in CocoImage.
- Renamed
CocoImage.delay
toCocoImage.imdelay
. The old name will be available as an alias, but will eventually be deprecated. - Import time is now much faster.
- default
check_aux
to True inCocoDataset.missing_images
- New
compress=True
argument todump
which will compress the kwcoco file before writing it. - Add CLI argument
--num_write
tokwcoco split
to write multiple splits.
- The kwcoco CLI is now strict (i.e. will error on unsupported arguments) by default. Set the
KWCOCO_LOOSE_CLI
environment variable to use the old behavior. - Added argcomplete support
- The
.fpath
attribute is no longer cast by default and is now allowed to be apathlib.Path
instance. In the future we may always convert this attribute to a pathlib object.
- Experimental support for reading / writing kwcoco files to / from zipfiles.
- Improved splitting in kwcoco split CLI. Better handling for video datasets. Category balance is now optional.
- Bug where specifying resolution to
delay
would not work if the channels did not exist.
- Add
role
property for annotations in the schema.
- Added
instance_id
for mmlab tracking - Fixed np.bool
- Fixed np.float
- Initial support to conform to mmlab coco specs
- Added ability to specify resolution in
CocoImage.delay
- Add resolution to the image / video schema.
- kwcoco validate now reports errors for all inputs
-
Modified signatures of
annots
andimages
to useannot_id
andimage_id
instead ofaid
andgid
. -
Removed mentions of "flicks" in timestamps. We are simply using ISO strings and unix times for now.
-
Updated timestamp details in the SPEC.
-
Updated descriptions and organization in the formal schema.
-
Require that names do not contain the "/" character in the formal schema.
- Initial support for PostgreSQL
- In CocoDataset.coerce the arg
sqlview
can now be given as a stringsqlite
orpostgresql
.
- Lookup on SQL database now handles json columns
- Taking non-existing channels from a single-image delayed node returns a concatenated node with existing and nan channels.
- Taking subchannels from a concatenated delayed node now groups discontinuous indices from the same underlying image.
channel_spec
,sensorchan_spec
,util.lazy_frame_backends
, andutil.delayed_ops
have all been ported to the newdelayed_image
package.
- Removed old
util_delayed_ops
- Fixed bug in vidshapes toydata code
- Respect masked arrays when doing delayed warp
- Handle change in kwimage demo data, doctest hashes changed
- The SQLView changed the "extra" key to "unstructured" and now implicitly removes it before being passed to the user.
- The multisensor data now contains a "sensor" key in the image dictionary.
- Can now pass
kwcoco.CocoDataset.coerce
sqlview=True
to attempt to grab an SQL view of the dataset.
- bug in coco subset when absolute=False
- Officially deprecate
kwcoco.util.util_delayed_poc
- Remove the 2 suffix on the official
delayed_ops
node names. - The delayed crop method can now simulated a padded slice.
- Changed ujson integration from opt-out to opt-in. Specify
KWCOCO_USE_UJSON
to use ujson.
- kwcoco will now use ujson to read json files by default if it is available.
- Added runtime deprecation noticed to deprecated functions
- Doctests now generate images for readthedocs.
- Ability to "undo warps" from delayed ops.
- Removed Jagged from
delayed_ops
- NaNs are now optimized like other leafs.
- Bug where delayed nan did not have the right warp signature.
- Bug in boids when speed was 0.
with_bundle
flag toiter_image_filepaths
- Bug in sensorchan spec
- Issue in reroot and toydata generation that used incorrect bundle-dpaths
- Fixed issue in key parsing where numbers greater than 10 failed.
- Added
fpath
as an option toCocoDataset.demo
, which is currently semi-stable. - The
finalize
method for a DelayedOperation2 now calls prepare and optimize beforehand by default, and also always returns an ndarray. - DelayedWarp2 will now default to nan fill values when warping floats
- Initial non-experimental version of SensorChanSpec with limited functionality
- Updated
kwcoco validate
cli tool with full options (and added checks for img attrs) - New Delayed operations with a more scalable design
- Fix compatibility issue with datasets that do not contain an annotations list
- Issue where the original coco polygons did not validate correctly in the schema.
- The new delayed operations now always return an arraylike
- FusedChannelSpec now has an
__eq__
method that returns True if the parsed reprs are the same.
- kwcoco subset can now accept dst as the second positional argument
- Fix issue in kwcoco subset where paths were invalid if written outside of the source kwcoco bundle.
- kwcoco validate now accepts
--require_relative
which causes non-portable kwcoco files with non-relative paths to fail. - kwcoco validate now checks for duplicate bands in the auxiliary dictionary.
- Better verbosity for demodata
- Experimental support for ujson
- Remove deprecated usage of cfgstr with ubelt.Cacher
- kwcoco stats output is modified
- CocoImage Affine transform
matrix
attribute is now always populated.
- CocoDataset.dump now uses
safer
withtemp_file=True
by default. - Initial steps to rename "auxiliary" to "assets" in a backwards compatible way.
- Bug in valid region warp
- Method to get the valid region for a CocoImage in image or video space
- CocoSqlDatabase is now lazy exposed at the top level in Python 3.7+
- Better SQL support in CocoImage
- Add maxlen in
path_sanitize
- Warp properties in CocoImage
- Stub out CocoAsset
- Can now load do delayed loading of data in auxiliary space. (throws an error if multiple auxiliary streams are mixed)
- issubset issuperset for FusedChannelSpec
- union for FusedChannelSpec and ChannelSpec
find_asset_obj
for CocoImage- Added quantization spec. CocoImage.delay will now return a dequantized image if quantization metadata is available.
- verbose flag to perterb function
- Added new exceptions
DuplicateAddError
andInvalidAddError
to better indicate why an add image/category/annotation failed.
- Combine measures can now take an explicit set of threshold bins to accumulate to.
- Bug in delayed with nodata
path_sanatize
tokwcoco.ChannelSpec
to make a path-safe suffix for a filename- Add experimental
_dataset_id
function tokwcoco.CocoDataset
for unique but human interpretable dataset identifiers.
- Conform now adds width / height to auxiliary objects
- Enhanced the
CocoImage.primary_asset
function - gdal reader now uses an auto nodata mode, somewhat experimental.
- Caching mechanism for model hashids
- Minor improvements to confusion measures.
- AP / ROC curves now report real/support using concise SI for large numbers
- Removed old Python2 constructs
- Modified multisensor to use concise channel codes
- Moved
LazyGDalFrameFile
tokwcoco.util.lazy_frame_backends
to experiment with different image subregion loaders
-
names kwarg to CocoDataset.images / videos
-
DelayedImage.finalize
now accepts anodata
argument, which handles invalid data pixels correctly under transformations. -
Toydata can now generate "multi-sensor" demodata.
-
CocoImage now returns None for
video
if it doesn't have one. -
BOIDS is now deterministic given a seed, which fixes toydata determinism
-
Fixed toydata bug where data previously only drawn on first channel
- Tweaked default toydata settings
- Added
add_auxiliary_item
toCocoImage
CocoImage.delay
can now handle the case whereimdata
is given directly in the image dictionary.Measures.combine
now has agrowth
andforce_bins
parameter that works somewhat better than the previousprecision
parameter.- Add measure combiner helper classes
- Support for detaching
CocoImage
from thedset
object (for multiprocessing)
- Split part of
confusion_vectors.py
into new fileconfusion_measures.py
- Added "channels" argument to kwcoco subset
- Bug in
delayed_load
when none of the requested channels exist
- Added more docs to demodata
- Moved eval CLI code to the cli module.
- Annotation outside the image bounds now render correctly in the toydata module.
- Fixed issue in FusedChannelSpec where a unnormalized getitem did not work correctly
- Ported channel related features to CocoImage
- vidshapes-msi is now an alias for vidshapes-multispectral
- Can now specify general channels for video toydata
- Split
kwcoco.demo.toydata
intokwcoco.demo.toydata_image
andkwcoco.demo.toydata_video
- Moved code from
coco_dataset
to_helpers
- Bug in
show_image
with MSI - Fixed bug in ObjectList1D.peek
- Moved Archive code to its own util module
- Extended channel slice syntax. Can now use "." to separate root from the slice notation.
- Removed debugging print statements
- Fixed issue with pickling category tree
- Fixed bugs with numel and zero channels
- Add
.images
toVideos
1D object. - Initial
copy_assets
behavior for kwcoco subset.
- Improved speed of repeated calls to FusedChannelSpec.coerce and normalize
- Fixed bug in delayed image where nans did not correctly change size when warped
- Fixed bug in delayed image where warps were not applied correctly to concatenated objects
- Initial implementation of shorthand channels
- Parameterized
max_speed
of toydata objects - Add
combine_kwcoco_measures
function - Add new API methods to ChannelSpec objects
- Reworking kwcoco channel spec backend implementation. Trying to maintain compatibility and insert warnings where things might change.
- Removed six from CocoDataset
- Added support for pathlib
- Fixed bug in union with relative paths
ChannelSpec
can now be coerced from aFusedChannelSpec
FusedChannelSpec
now implements the__set__
interface
- Fixed bug when
track_id
is given toadd_annotation
- Fixed but in
delayed_load
where requested channels were returned in the wrong order, or with incorrect data. - Bug in
delayed_load
where nans did not resize properly
- Can now specify
-frames{num}
in demo names to control number of frames in each video
- In detection metrics, annotations now get a default score of 1.0 if not provided.
- In detection metrics, fixed AUC to report 1.0 when detections are perfect
- Added
vidid
argument toCocoDataset.images
1D object mixin. - Added
trackid
argument toCocoDataset.annots
1D object mixin.
perterb_coco
now correctly perturbs segmentations
- Changed argument name from
gsize
toimage_size
in toydata methods. Backwards compatibility is preserved via kwargs for now, butgsize
will be deprecated in the future.
- CocoSQLDataset.coerce
kwcoco subset
CLI now has a--select_images
and--select_videos
option that takes a json query.- Add initial implementation of
DelayedChannelStack.take_channels
, returns nans if the channel is not available.
- Made URIs for CocoSQLDataset slightly more robust
- kwcoco show now defaults
show_labels
to False
- Added "domain net" to
kwcoco grab
. - Added "spacenet7" to
kwcoco grab
.
- Fixed issue in
add_annotation
where if failed to add keypoints, and segmentations.
-
CocoDataset.index.trackid_to_aids
is a new index that maps a track-id to all annotations assigned to that track. -
kwcoco grab
CLI for downloading and converting standard datasets -
CocoDataset.delayed_load
now has a limited ability to specify a list of channels to load, and if it is returned in image or video space.
kwcoco.add_annotation
can now accept and automatically convert kwimage data structures for segmentation and keypoints.
- Channels in a base image can now be None.
validate
now prints which field caused the issue
- Updated README
- Code cleanup
- More docs
- Fixed error with empty union
- Fixed error in
delayed_load
- Experimental
delayed_load
method for an image
- Fixed
cats_per_img
in kwcoco stats. - Fixed
grab_camvid
. - Fixed legacy
img_root
key not being respected.
- Made bezier dependency optional
- stats now returns video stats
- Added
channel_spec.py
as a helper and documentation - Auxiliary images now have
warp_aux_to_img
property - Base images now have
warp_img_to_vid
property
- Removed
base_to_aux
transform in favor of inversewarp_aux_to_img
- Fixed error in kwcoco conform documentation
- The conform operation now warns if the segmentation cannot be converted to a polygon
- The
dset.index.vidid_to_gids
is now guaranteed to always return image ids ordered by their frame index. dset.view_sql
now has amemory
kwarg that will force the database to live in memory.- The SQLite backend now agrees with the dict backend and does not map None to values for image names and file names.
kwcoco.utils.util_slice
usekwarray.util_slice
instead.
- Fixed numpy warning by using
int
instead ofnp.int
. - Issue in rename categories with non-deterministic behavior
- Reference gdal from the
osgeo
package - Fixed "name" attribute in sqlview.
- Implemented the
name_to_video
index - annToMask and annToRLE in the
compat_dataset
now work like the original.
- Several deprecated APIs
kwcoco.toydata
andkwcoco.toypatterns
, usekwcoco.demo.toydata
andkwcoco.demo.toypatterns
instead.remove_all_images
andremove_all_annotations
(use clear variants instead)
kwcoco coco_subset
can now take a list of image-ids explicitly.- Separate the vectorized ORM-like objects out of
coco_dataset
and into their owncoco_objects
module.
-
The image schema now has two modalities. The normal
id
+file_name
still exists, but now we can doid
+name
+auxiliary
without a defaultfile_name
, which allows us to better handle multispectral images. -
The
name
is a new primary text key for images. -
In the SQL view of the database, non-schema properties are now stored in "extra" instead of "foreign". (I confused "foreign keys" for "additionalProperties")
- Add
name
property to theimage
. - Add
kwcoco.Subset
create a subset of a coco file based on filter criteria - Add
kwcoco.Subset
create a subset of a coco file based on filter criteria - Experimental interactive mode to kwcoco show
- Start to merge cli and class functionality.
-
Fixed the default bundle root to be the cwd instead of
untitled_kwcoco_bundle
. -
CocoDataset.union
will now correctly insert prefixes into file names such that they are relative to the common bundle directory. -
Bug in
rename_categories
- Introduced the concept of a bundled dataset
- Added
kwcoco conform
script to standardize / make coco files compliant withpycocotools
. - Added
--image_attrs=True
tokwcoco stats
. - Added
AbstractCocoDataset
base class forCocoDataset
andCocoSqlDataset
. - Added
examples
subdirectory for answers to FAQ - Added
kwcoco validate
script to check that json and assets exist. - Added
conform
method to the main coco dataset
CocoDataset.subset
will now only return the videos that are supported by the chosen images.
- Add
ascii_only
keyword to clf-report to disable unicode glyphs - Add
ASCII_ONLY
environment variable to disable unicode glyphs - Initial implementation for
CocoSqlDatabase
- Fix bug in
show_image
when segmentation is None
- Add
remove_video
method to CocoDataset - kwcoco show can now show auxiliary data
-
CocoDataset.union
now will remap track-ids assuming all input datasets are disjoint by default. -
Fixed issues in metrics classification report
- Standalone algorithm for BOID positions
- Make kwplot, matplotlib, and seaborn optional
- Fixed bug in
draw_threshold_curves
when measures are empty.
- Better support for pycocotools
- Better evaluation code
- Better auxiliary data
- Tweaks in CLI stats script
kwcoco.kw18.KW18.to_coco
can now acceptimage_paths
andvideo_name
- Yet more change to
reroot
, this needs to get reviewed and fixed. - Fix spelling of auxillary to auxiliary
- Better auxiliary channel support in toydata
- Added version flag to the CLI
- Fixes to
ensure_json_serializable
for edge cases and recursion depth
- Removed
pred_raw
from ConfusionVectors
- Ported the non-torch components of CategoryTree from ndsampler to kwcoco.
- This helps kwcoco work better as a standalone package.
- Union now works with videos
- New formal schema with jsonschema validation support
- Add
autobuild
kwarg toCocoDset.subset
- Removed optional dependency on ndsampler, functionality should now be available without ndsampler.
- the
_check_index
method now works correctly. add_image
withvideo_id
now works even if the video doesn't exist yet.add_annotation
no longer fails ifcategory_id
is None.
- Added
compat_dataset.COCO
which supports 99% of the originalpycocotools
API. - Added
kwcoco modify_categories
CLI which supports renaming and removal of categories. - New
random_video_dset
demo function to generate toydata in video sequences. Added new tooling for this. STATUS: INCOMPLETE - Add new video-based APIs to the CocoDataset based on the spec. STATUS: incomplete
- Fix bug in find_representative_images.
- Basic KW18 read/write/from-COCO support
- Fixed json serialization error when using perterb_coco.
- ported metrics from netharn, might be moved in the future.
- Add
perterb_coco
- Add
CocoEvaluator
and CLIkwcoco eval
- stats CLI can now accept multiple datasets.
- ported
data.grab_voc
anddata.grab_camvid
from netharn. - Add
kwcoco.coco_evaluator.CocoEvaluator
- Add
safe
kwarg toCocoDataset.reroot
- Python2 error with json dump
- Example usage section in CLI help
- Moved toydata from ndsampler to kwcoco
- Multipolygon segmentations are now displayed correctly.
- Tracebacks on errors are now displayed.
- Initial version