Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating ML benchmarks #154

Open
wants to merge 140 commits into
base: mo_experiments
Choose a base branch
from
Open

Updating ML benchmarks #154

wants to merge 140 commits into from

Conversation

Neeratyoy
Copy link
Collaborator

  • Added option to record Learning Curves
  • Bug fix: new tabular data collected by training on train set and evaluating on validation set

Neeratyoy added 30 commits June 23, 2021 20:32
@Neeratyoy
Adding sample RF space for tabular collection design
b2155dd
@Neeratyoy
Placeholder SVM benchmark to interface tabular data collection
ce405e6
@Neeratyoy
Writing common ML benchmark class for tabular collection
2ef3af8
@Neeratyoy
Adding placeholder for HistGradientBoostedClassifier
61b6963
@Neeratyoy
Minor code cleaning
a5d0217
@Neeratyoy
Reformatting output dict + option to add more metrics
3def203
@Neeratyoy
Removing redundant import
750cc7d
@Neeratyoy
Decoupling storage of costs for each metric
e7665e6
@Neeratyoy
Including test scores in objective
47fe4cd
@Neeratyoy
Documenting the structure of information in each fn eval.
2d085ec
@Neeratyoy
Some decisions on lower bound for subsample fidelity
2da9d5c
@Neeratyoy
AbstractBenchmark update for fidelity option + including XGBoost
751d2e9
@Neeratyoy
Adding sample RF space for tabular collection design
3f84afb
@Neeratyoy
Placeholder SVM benchmark to interface tabular data collection
09b296a
@Neeratyoy
Writing common ML benchmark class for tabular collection
af4f593
@Neeratyoy
Adding placeholder for HistGradientBoostedClassifier
df2462d
@Neeratyoy
Minor code cleaning
4d1d2d6
@Neeratyoy
Reformatting output dict + option to add more metrics
299e592
@Neeratyoy
Removing redundant import
c46321d
@Neeratyoy
Decoupling storage of costs for each metric
17f6634
@Neeratyoy
Including test scores in objective
7de891f
@Neeratyoy
Documenting the structure of information in each fn eval.
ec316c3
@Neeratyoy
Some decisions on lower bound for subsample fidelity
e7f69b9
@Neeratyoy
AbstractBenchmark update for fidelity option + including XGBoost
edb3e7f
@Neeratyoy
Merge branch 'thesis-paper' of https://github.com/Neeratyoy/HPOBench …
642027b 
…into thesis-paper
@Neeratyoy
Option to load data splits from disk
9e907e6
@Neeratyoy
Reordering data load to work for different cases
f0d4f36
@Neeratyoy
Updating source of SVM HP range
dbeae7c
@Neeratyoy
Adding Tabular Benchmark class
f277a2e
@Neeratyoy
Adding TabularBenchmark interface + easy import
60d5646
@Neeratyoy Neeratyoy requested a review from PhMueller May 30, 2022 13:11
@Neeratyoy Neeratyoy changed the base branch from master to development May 30, 2022 14:03
@Neeratyoy Neeratyoy changed the base branch from development to mo_experiments June 4, 2022 19:26
Neeratyoy added a commit that referenced this pull request Jun 6, 2022
@Neeratyoy
Simulating PR #154 merge
08e50c3
Neeratyoy and others added 10 commits August 31, 2022 14:59
@Neeratyoy
Updating container version used for experiments
dfa2034
@PhMueller @dwoiwode
HPOBench with python >= 3.10 (#162)
5889a04 
Make installable for Python 3.10.6
* Add 3.10 to test pipeline
* Remove upper Python version limit
* Change python-versions to strings
* Surround action python-version with quotation marks

* Install paramnet dependencies only for Python <= 3.9
* Disable local paramnet test for Python > 3.9

Co-authored-by: Dominik Woiwode <[email protected]>
@PhMueller
Speed up CI testing (#163)
45b1eb0 
* Speed up tests by disabling slow tests.
@PhMueller
Refactor PR (#164)
3b2fa7b 
* Refactor: Rename AbstractBenchmark Class
@PhMueller
Nas201 - v0.0.6 (#165)
ebff935 
* Update 201 version 0.0.6

- scale the misclassification rate form 0 to 100 -> 0 and 1
- Update test cases
- link to the correct container version
- Fix Nas201 Test Cases
- remove a Github action trigger. It started all test cases twice. 
- Fix GithubActions: Install correct xgboost version for 3.10
@PhMueller
Nas101 + Nas1shot1 v0.0.5 - Multi Objective (#166)
4414e3d 
* Nas101 v.0.0.5
* Nas1shot1 v.0.0.5
* Test Cases for 101
@PhMueller
MO CNN and MO Adult v0.0.2 (#167)
47f8b5b 
Update to 0.0.2
@PhMueller
Towards compatibility with windows os. (#170)
46f719d
@PhMueller @ayushi-3536 @dwoiwode
Raw YAHPO Benchmarks (#153)
236e542 
* Yahpo Raw: You can run fair/iml/rbv2 experiments from yahpo. 

The installation is quite complicated and described in the recipe. 

* Introduce YAHPO Surrogate V0.0.2
It also goes hand in hand with an update for the yahpo surrogate benchmarks. 

---------

Co-authored-by: ayushi-3536 <[email protected]>
Co-authored-by: ayushi-3536 <[email protected]>
Co-authored-by: Dominik Woiwode <[email protected]>
@KEggensperger
Merge branch 'development' into LCs
36acf0d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PhMueller PhMueller Awaiting requested review from PhMueller

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Neeratyoy @PhMueller @KEggensperger