Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Nov 7, 2025

• Merge pull request #2 from howard-yen/main

  Show description for e51355c
  CodeCreator

  authored
  e51355c

  Copy full SHA for e51355c

Commits on Nov 5, 2025

• update README to reflect updated argument name

  howard-yen

  committed
  295e710

  Copy full SHA for 295e710

• add support for loading gz files for jsonl

  howard-yen

  committed
  58e9ebd

  Copy full SHA for 58e9ebd

• merge

  howard-yen

  committed
  88be4b0

  Copy full SHA for 88be4b0

Commits on Oct 17, 2025

• bump to 0.5

  CodeCreator

  committed
  95dbe93

  Copy full SHA for 95dbe93

• add __init__ files

  CodeCreator

  committed
  cdffd0a

  Copy full SHA for cdffd0a

• bump to 0.4

  CodeCreator

  committed
  4434c2c

  Copy full SHA for 4434c2c

• bump to 0.3 && fix packages

  CodeCreator

  committed
  973bd11

  Copy full SHA for 973bd11

Commits on Sep 10, 2025

• Add parquet/pyarrow support (w/o hf)

  CodeCreator

  committed
  31c404b

  Copy full SHA for 31c404b

Commits on Aug 19, 2025

• Bump version to v0.2

  CodeCreator

  committed
  56f591b

  Copy full SHA for 56f591b

• load s3 mosaic dataset and make hf optional

  CodeCreator

  committed
  164d624

  Copy full SHA for 164d624

Commits on Apr 9, 2025

• update readme with example

  howard-yen

  committed
  4025820

  Copy full SHA for 4025820

Commits on Apr 2, 2025

• Update README.md

  CodeCreator

  authored
  81b94a7

  Copy full SHA for 81b94a7

• Update README.md

  CodeCreator

  authored
  18c6e32

  Copy full SHA for 18c6e32

• Update README.md

  CodeCreator

  authored
  3645250

  Copy full SHA for 3645250

• Update README.md

  CodeCreator

  authored
  b28e0b9

  Copy full SHA for b28e0b9

• Update README

  CodeCreator

  committed
  8d118ec

  Copy full SHA for 8d118ec

• Update README.md

  CodeCreator

  authored
  7d421ac

  Copy full SHA for 7d421ac

• Update README.md

  CodeCreator

  authored
  cde52e9

  Copy full SHA for cde52e9

• fix csv

  howard-yen

  committed
  5d9b303

  Copy full SHA for 5d9b303

• add documentation in readme

  howard-yen

  committed
  c3ae514

  Copy full SHA for c3ae514

Commits on Apr 1, 2025

• update tsv

  howard-yen

  committed
  6b88c06

  Copy full SHA for 6b88c06

• support for loading tsv and csv

  howard-yen

  committed
  b9b5317

  Copy full SHA for b9b5317

Commits on Mar 5, 2025

• Integrate shuffling to packing (for ease of use)

  CodeCreator

  committed
  fc1c854

  Copy full SHA for fc1c854

• Rename option -> (don't break up documents into more than one chunk)

  CodeCreator

  committed
  eece242

  Copy full SHA for eece242

• Packing: also pack extra fields (like attention masks -- note: need to fix eos/bos)

  CodeCreator

  committed
  535577c

  Copy full SHA for 535577c

• Llama tokenizer: return assistant masking

  CodeCreator

  committed
  2e453c3

  Copy full SHA for 2e453c3

• tokenize: apply chat format (llama only for now)

  CodeCreator

  committed
  a17470d

  Copy full SHA for a17470d

Commits on Feb 19, 2025

• Add setup directions

  CodeCreator

  committed
  3cd0b65

  Copy full SHA for 3cd0b65

• Update README.md

  CodeCreator

  authored
  be0db56

  Copy full SHA for be0db56

• Update README.md

  CodeCreator

  authored
  6067197

  Copy full SHA for 6067197

• Fix README.md header

  CodeCreator

  authored
  0de941d

  Copy full SHA for 0de941d

• Update README.md header

  CodeCreator

  authored
  87d27ba

  Copy full SHA for 87d27ba

• Update README title

  CodeCreator

  committed
  335e053

  Copy full SHA for 335e053

• Update README

  CodeCreator

  committed
  c297847

  Copy full SHA for c297847

Pagination

Previous

Next