宇健 何 posted an update — Aug 25, 2020 04:47 PM EDT

Updates（2020.08.25）

This update implemented two commonly used ensemble methods (bagging, adaboost). However, the related codes are still W.I.P, so the main purpose of this update is to show the potential of carefree-learn:

ensemble = cflearn.Ensemble(TaskTypes.CLASSIFICATION, config)
results = ensemble.adaboost(train_file)
predictions = results.pattern.predict(test_file)

Log in or sign up for Devpost to join the conversation.

宇健 何 posted an update — Aug 15, 2020 12:47 PM EDT

Updates（2020.08.16）

This update is mainly about miscellaneous fixes, but I've also introduced a toy example to reveal the power of carefree-learn - the famous Titanic competition!

Here are the source codes:

import os
import cflearn

from cfdata.tabular import *

file_folder = os.path.dirname(__file__)


def test():
    train_file = os.path.join(file_folder, "train.csv")
    test_file = os.path.join(file_folder, "test.csv")
    data_config = {"label_name": "Survived"}
    hpo = cflearn.tune_with(
        train_file,
        model="tree_dnn",
        temp_folder="__hpo__",
        task_type=TaskTypes.CLASSIFICATION,
        data_config=data_config,
        num_parallel=0
    )
    results = cflearn.repeat_with(
        train_file,
        **hpo.best_param,
        models="tree_dnn",
        temp_folder="__repeat__",
        num_repeat=10, num_jobs=0,
        data_config=data_config
    )
    ensemble = cflearn.EnsemblePattern(results.patterns["tree_dnn"])
    predictions = ensemble.predict(test_file).ravel()
    x_te, _ = results.transformer.data.read_file(test_file, contains_labels=False)
    id_list = DataTuple.with_transpose(x_te, None).xT[0]
    # Score : achieved ~0.79
    with open("submissions.csv", "w") as f:
        f.write("PassengerId,Survived\n")
        for test_id, prediction in zip(id_list, predictions):
            f.write(f"{test_id},{prediction}\n")


if __name__ == '__main__':
    test()

As you can see, carefree-learn doesn't need explicit data-preprocessing - it can take files as inputs and predict with files directly! More over, some common practises, such as hyper parameter tuning (cflearn.tune_with) and ensembling (cflearn.repeat_with & cflearn.EnsemblePattern), can be completed in a few lines of codes. These APIs also hide some other common practises (such as cross validation) under the hood, so the final performance is quite promising (I can achieve ~0.79 and the best one achieved 0.81+, which is almost the SOTA performance among other (more complicated) neural network solutions 1 2 3 4).

Log in or sign up for Devpost to join the conversation.

宇健 何 posted an update — Jul 31, 2020 10:46 PM EDT

Updates (2020.08.01)

Experiments

Experiments is much more powerful and much easier to use now:

import cflearn
import numpy as np

from cfdata.tabular import *

def main():
    x, y = TabularDataset.iris().xy
    experiments = cflearn.Experiments()
    experiments.add_task(x, y, model="fcnn")
    experiments.add_task(x, y, model="fcnn")
    experiments.add_task(x, y, model="tree_dnn")
    experiments.add_task(x, y, model="tree_dnn")
    results = experiments.run_tasks(num_jobs=2)
    # {'fcnn': [Task(fcnn_0), Task(fcnn_1)], 'tree_dnn': [Task(tree_dnn_0), Task(tree_dnn_1)]}
    print(results)
    ms = {k: list(map(cflearn.load_task, v)) for k, v in results.items()}
    # {'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
    print(ms)
    # experiments could be saved & loaded easily
    saving_folder = "__temp__"
    experiments.save(saving_folder)
    loaded = cflearn.Experiments.load(saving_folder)
    ms_loaded = {k: list(map(cflearn.load_task, v)) for k, v in loaded.tasks.items()}
    # {'fcnn': [FCNN(), FCNN()], 'tree_dnn': [TreeDNN(), TreeDNN()]}
    print(ms_loaded)
    assert np.allclose(ms["fcnn"][1].predict(x), ms_loaded["fcnn"][1].predict(x))

if __name__ == '__main__':
    main()

We can see that experiments.run_tasks returns a bunch of Tasks, which can be easily transfered to models through cflearn.load_task.

It is important to wrap the codes with main() on some platforms (e.g. Windows), because running codes in parallel will cause some issues if we don't do so. Here's an explaination.

Benchmark

Benchmark class is implemented for easier benchmark testing:

import cflearn
import numpy as np

from cfdata.tabular import *

def main():
    x, y = TabularDataset.iris().xy
    benchmark = cflearn.Benchmark(
        "foo",
        TaskTypes.CLASSIFICATION,
        models=["fcnn", "tree_dnn"]
    )
    benchmarks = {
        "fcnn": {"default": {}, "sgd": {"optimizer": "sgd"}},
        "tree_dnn": {"default": {}, "adamw": {"optimizer": "adamw"}}
    }
    msg1 = benchmark.k_fold(3, x, y, num_jobs=2, benchmarks=benchmarks).comparer.log_statistics()
    """
    ~~~  [ info ] Results
    ===============================================================================================================================
    |        metrics         |                       acc                        |                       auc                        |
    --------------------------------------------------------------------------------------------------------------------------------
    |                        |      mean      |      std       |     score      |      mean      |      std       |     score      |
    --------------------------------------------------------------------------------------------------------------------------------
    |    fcnn_foo_default    |    0.780000    | -- 0.032660 -- |    0.747340    |    0.914408    |    0.040008    |    0.874400    |
    --------------------------------------------------------------------------------------------------------------------------------
    |      fcnn_foo_sgd      |    0.113333    |    0.080554    |    0.032780    |    0.460903    |    0.061548    |    0.399355    |
    --------------------------------------------------------------------------------------------------------------------------------
    |   tree_dnn_foo_adamw   | -- 0.833333 -- |    0.077172    | -- 0.756161 -- | -- 0.944698 -- | -- 0.034248 -- | -- 0.910451 -- |
    --------------------------------------------------------------------------------------------------------------------------------
    |  tree_dnn_foo_default  |    0.706667    |    0.253684    |    0.452983    |    0.924830    |    0.060007    |    0.864824    |
    ================================================================================================================================
    """
    # save & load
    saving_folder = "__temp__"
    benchmark.save(saving_folder)
    loaded_benchmark, loaded_results = cflearn.Benchmark.load(saving_folder)
    msg2 = loaded_results.comparer.log_statistics()
    assert msg1 == msg2

if __name__ == '__main__':
    main()

Misc

• Integrated trains.
• Integrated Tracker from carefree-toolkit.
• Integrated native amp from PyTorch.
• Implemented FocalLoss.
• Implemented cflearn.zoo.

---

• Introduced CI.
• Fixed some bugs.
• Simplified some APIs.
• Optimized some default settings.

What's next

I've already done some experiments on some benchmark datasets with Benchmark and already achieved satisfying performance. However, large scale benchmark testing is not done yet, limited by my lack of GPU cards XD

So the next step is to do large scale benchmark testing and optimize carefree-learn's performance, in a much more general way.

In the mean time, I'll do some research and implement some SOTA methods on tabular datasets (e.g. Deep Sparse Network, β-LASSO MLP, ...)

And, as always, bug fixing XD

Log in or sign up for Devpost to join the conversation.

宇健 何 started this project — Jul 05, 2020 04:24 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.