Plots diff for different file paths in current workspace

[dberenbaum]

Users may run similar experiment variants in different directories or paths (especially now that foreach syntax makes this structure easy to set up). They may want to do a plot comparing between those paths rather than between git references.

This could borrow from the git diff --no-index syntax (https://mirrors.edge.kernel.org/pub/software/scm/git/docs/git-diff.html). For example:

dvc plots diff --no-index model1_roc.tsv model2_roc.tsv

To keep this simple, I'll ignore the implications for other commands (params/metrics) for now 😄.

[dberenbaum]

@pared Thoughts on this as a solution to one of the questions raised by @mnrozhkov?

[pared]

@dberenbaum I think that could be a proper solution.

That brings up a question of how to approach diffing "across revisions"/"between different files". Currently, our API is built around the idea that we compare the same files between revisions. I think that we need to

1. rethink how the API is structured so that we can provide information "I want to compare X_file from X_rev with Y_file from Y_rev and Z_file from Z_revision" - inside DVC, which could probably be done right away.
2. rethink how we are providing the diff arguments from CLI, as the current one has this idea of comparing files with the same name build in.

Providing a solution to 2 seems like breaking change from the user perspective, so we might need to consider (until 3.0) having current API and providing --no-index option that would "switch" the command to diff different files, but only on the same branch.

Also, there is a question of whether we want to provide the solution to it (comparing different files from different revisions) at all. I think it's hard to design CLI interaction that would allow us this functionality and be pleasant to use at the same time.

[dberenbaum]

1. rethink how the API is structured so that we can provide information "I want to compare X_file from X_rev with Y_file from Y_rev and Z_file from Z_revision" - inside DVC, which could probably be done right away.

Also, there is a question of whether we want to provide the solution to it (comparing different files from different revisions) at all

Hmm, good point. I'm not sure if that's a likely enough scenario to be worth the UI headache, but it does seem odd to leave it out if all the functionality is there. I guess DVC could always keep following the Git syntax: https://stackoverflow.com/questions/43041882/git-how-to-diff-two-different-files-in-different-commits 🤷 .

[pared]

@dberenbaum makes sense!

[dberenbaum]

Do you really think so? I'm less certain, since it's new and obscure syntax not used elsewhere. Keep in mind that whatever we do with plots will probably need to implemented across other output types. I think it's probably fine to either borrow the Git syntax or leave out this use case.

[pared]

@dberenbaum

makes sense!

Was applying to revision:target format - it does seem logical and easy enough to use. However there is one ceveat (besides the fact that we would basically need to restructure whole logic for outputs collection): we do support : in output names:

#!/bin/bash

rm -rf repo
mkdir repo

pushd repo
git init --quiet
dvc init --quiet

git add -A
git commit -am "init"

echo data >> data
dvc add data

dvc run -n train -d data -o "a:c" "cat data >> 'a:c'"

Which just adds up naming restrictions on outputs on top of other things we would need to do. So, this use case (cross-revision, cross-files diff command), if it will ever be implemented, should be left for 3.0.

As to solving current issue: It seems that we do agree that, for now --no-index is the way to go, right?

[dberenbaum]

As to solving current issue: It seems that we do agree that, for now --no-index is the way to go, right?

👍

I also realized that this looks like a duplicate of #5074. If you want to close it out in favor of that one, feel free.

[pared]

Here is the most of discussion, lets not split the conversation

[mnrozhkov]

@pared Thoughts on this as a solution to one of the questions raised by @mnrozhkov?

I'm afraid it's not :) Could you remind me what was the question? This seems like a workaround in current paradigm of plotting functionality in DVC.

Here are few concerns:

• such an approach push me to generate and save few modelN_roc.tsv files which is cumbersome
• in case of GridSearch with dozens of combinations it seems unfeasible solution
• in case I need to compare two or more models I would prefer to save a separate plot for this purpose (maybe in image)

[dberenbaum]

I'm afraid it's not :) Could you remind me what was the question?

DVC plots diffs currently rely on comparing the same filepath between different commits. I thought you had asked about a workflow where outputs are spread across multiple filepaths/directories in the same commit/workspace. If we misunderstood, we can consider closing this issue and continue discussing other concerns elsewhere.

[pared]

@mnrozhkov how does data structure look like in case of, for example GridSearch? Would you dump all results into a single file?

[dberenbaum]

Yes, I think this is getting a bit off-topic for this issue, but would getting data from many experiments into one file that you could use to generate various plots (or do any other analysis) address some of your concerns @mnrozhkov?

[mnrozhkov]

Hi @dberenbaum, @pared ! Sorry, I've missed notifications on your comments. I try to remember which use case we discussed that may lead to this issue. Probably, this is due to the customer use case with 2 models (on different feature sets):

• outputs for both models/pipelines are spread across multiple filepaths/directories in the same commit/workspace
• however, this is a legacy pipeline that we have already converted into a single pipeline with 2 different configs
• so, we may consider them them as separate experiments

The main request is - how to plots show/diff plots with multiple data series (like GridSearch)?
The data for GridSearch may look like this (in .csv):

max_depth,gini,lr=0.04,lr=0.07,lr=0.1
1,0.06261619717587696,0.08635409877237366,0.09777655339498081,0.10466567845566543
2,0.10712440926869116,0.15084024958294973,0.17314072866726993,0.18457377898108485
3,0.14396704110978598,0.2089246501682127,0.23467739169156032,0.2419810272380536
4,0.17842031439831052,0.24784909569548286,0.2720820535408013,0.28595938909499896
5,0.21265548218105557,0.28633725278198097,0.3046207648619761,0.30921103916741965
6,0.22307857499271133,0.2957091262006297,0.311709119889805,0.3103388383504939
7,0.2290581783322998,0.2973207556273978,0.3089473864643941,0.3064052115486884

At this moment, the only options is to save each plot separately. And try to compare them.

This issue #5693 actually could help with this @dberenbaum. However, it seems not the right directions due to reasons discussed above.

In my opinion, we need to think about

1. support for multi data series in one plot (like GridSearch)
2. support images

[dberenbaum]

In my opinion, we need to think about

1. support for multi data series in one plot (like GridSearch)

2. support images

👍 Yes, I agree. These are not likely to be implemented in the short-term, however, so a couple thoughts on how to deal with each:

1. In the example above, it seems that you want to visualize 21 experiments (7 max depths X 3 learning rates, assuming the gini column is a typo). This is an issue of both multi-series plots and having data from multiple experiments in one output file. For now, you could get the data from dvc exp show, do a crosstab/pivot table, and then plot however you like. The only problem is that there's no easily parseable output format for dvc exp show just yet. cc @pmrowla
2. There's no reason users can't generate images instead of using dvc plots, even if there's no dvc-specific support for them. You could even diff them with various Git tools. If there are particular needs you have in mind, feel free to follow up.

[mnrozhkov]

Yes, I think this is getting a bit off-topic for this issue, but would getting data from many experiments into one file that you could use to generate various plots (or do any other analysis) address some of your concerns @mnrozhkov?

Probably yes. In this case we need to think about such file structure and this file need to be changed (append new experiment data)by few tasks/pipelines. This seems not a good pattern.

Another issue for such case is usability. For my personal projects, I really don't like to do plots generation in multi-steps scenario:

1. Save data for plots (at dvc repro step)
2. Run dvc plots show/diff to generate plots.

I could understand, that this is due to DVC's language agnostic nature. I would like to raise discussion to have a python API to generate plots (like Altair).

[dberenbaum]

I agree that the current approach is clunky, having to configure and generate plots separately.

Using Altair or some other plotting library to visualize plots within Python makes sense, but I think it stretches outside of the scope of the core DVC project a bit. Besides being a Python API, Altair is also a dataviz library, and DVC is otherwise a command-line tool (the existing Vega plots already stretch the core functionality). There are also users who will want some other plotting library instead. That's why I suggest that DVC might support returning the underlying plots data and enable any kind of dataviz to be built on top of it. You might want to check out the plots use case in https://www.notion.so/iterative/Unified-show-diff-commands-df3c941296e84c6ba6c2fab1c3a55f7e and provide feedback there.