thomaskim1130/stella_en_1.5B_v5-FinanceRAG

SentenceTransformer based on dunzhang/stella_en_1.5B_v5

This is a sentence-transformers model finetuned from dunzhang/stella_en_1.5B_v5. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: dunzhang/stella_en_1.5B_v5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 1536, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 1536, 'out_features': 1024, 'bias': True, 'activation_function': 'torch.nn.modules.linear.Identity'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Instruct: Given a web search query, retrieve relevant passages that answer the query.\nQuery: Title: \nText: | _id | q83c2a93e |\n| title |  |\n| text | in 2006 what was the total amount authorized by the board of directors authorized for the repurchase of shares in billions\n\nin 2006 total amount authorized by board of directors authorized for repurchase of shares in billions\n\n\n',
    'Title: \nText: | _id | d828efc52 |\n| title |  |\n| text | Item 1B.\nUnresolved Staff Comments.\nNone.\nItem 2.\nProperties.3M\x80\x99s general offices, corporate research laboratories, and certain division laboratories are located in St.  Paul, Minnesota.\nIn the United States, 3M has nine sales offices in eight states and operates 74 manufacturing facilities in 27 states.\nInternationally, 3M has 148 sales offices.\n\nInternationally, 3M has 148 sales offices.\nThe Company operates 93 manufacturing and converting facilities in 32 countries outside the United States.3M owns substantially all of its physical properties.3M\x80\x99s physical facilities are highly suitable for the purposes for which they were designed.\nBecause 3M is a global enterprise characterized by substantial intersegment cooperation, properties are often used by multiple business segments.\nItem 3.\nLegal Proceedings.\n\nItem 3.\nLegal Proceedings.\nDiscussion of legal matters is incorporated by reference from Part II, Item 8, Note 13, \x80\x9cCommitments and Contingencies,\x80\x9d of this document, and should be considered an integral part of Part I, Item 3, \x80\x9cLegal Proceedings.\n\x80\x9d Item 4.\nSubmission of Matters to a Vote of Security Holders.\nNone in the quarter ended December 31, 2007.\nPART II Item 5.\nMarket for Registrant\x80\x99s Common Equity, Related Stockholder Matters and Issuer Purchases of Equity Securities.\n\nEquity compensation plans\x80\x99 information is incorporated by reference from Part III, Item 12, \x80\x9cSecurity Ownership of Certain Beneficial Owners and Management and Related Stockholder Matters,\x80\x9d of this document, and should be considered an integral part of Item 5.\nAt January 31, 2008, there were approximately 121,302 shareholders of record.3M\x80\x99s stock is listed on the New York Stock Exchange, Inc.  (NYSE), the Chicago Stock Exchange, Inc. , and the SWX Swiss Exchange.\n\nCash dividends declared and paid totaled $.48 per share for each quarter of 2007, and $.46 per share for each quarter of 2006.\nStock price comparisons follow:\n\nItem 1B.\n Unresolved Staff Comments.\n None.\n Item 2.\n Properties. 3M\x80\x99s general offices corporate research laboratories division laboratories located in St. Paul, Minnesota.\n United States 3M has nine sales offices in eight states operates 74 manufacturing facilities in 27 states.\n Internationally 3M has 148 sales offices.\n\nInternationally 3M has 148 sales offices.\n Company operates 93 manufacturing converting facilities in 32 countries outside United States. 3M owns all physical properties. 3M\x80\x99s physical facilities suitable for purposes designed.\n 3M global enterprise substantial intersegment cooperation properties used by multiple business segments.\n Item 3.\n Legal Proceedings.\n\nItem 3.\n Legal Proceedings.\n Discussion of legal matters incorporated reference from Part II, Item 8, Note 13, \x80\x9cCommitments and Contingencies document integral part of Part I, Item 3, \x80\x9cLegal Proceedings.\n Item 4.\n Submission of Matters to Vote of Security Holders.\n None in quarter ended December 31, 2007.\n PART II Item 5.\n Market for Registrant\x80\x99s Common Equity Related Stockholder Matters Issuer Purchases of Equity Securities.\n\nEquity compensation plans\x80\x99 information incorporated reference from Part III, Item 12, \x80\x9cSecurity Ownership of Certain Beneficial Owners Management Related Stockholder Matters document integral part of Item 5.\n At January 31, 2008, approximately 121,302 shareholders of record. 3M\x80\x99s stock listed on New York Stock Exchange. (NYSE), Chicago Stock Exchange, Inc. SWX Swiss Exchange.\n Cash dividends declared paid totaled $. 48 per share for each quarter of 2007, $. 46 per share for each quarter of 2006.\n\nStock price comparisons follow\n\n\n',
    'Title: \nText: | _id | d815ddb0a |\n| title |  |\n| text | Income Taxes 2017 Tax Act: The President signed U. S.  tax reform legislation (\x80\x9c2017 Tax Act\x80\x9d) on December 22, 2017, which is considered the enactment date.\nThe 2017 Tax Act includes a broad range of provisions, many of which significantly differ from those contained in previous U. S.  tax law.\nChanges in tax law are accounted for in the period of enactment.\nAs such, our 2017 consolidated financial statements reflect the immediate tax effect of the 2017 Tax Act.\n\nThe 2017 Tax Act contains several key provisions including, among other things: ?\na one-time tax on the mandatory deemed repatriation of post-1986 untaxed foreign earnings and profits (E&P), referred to as the toll charge; ?\na reduction in the corporate income tax rate from 35 percent to 21 percent for tax years beginning after December 31, 2017; ?\n\nthe introduction of a new U. S.  tax on certain off-shore earnings referred to as global intangible low-taxed income (GILTI) at an effective tax rate of 10.5 percent for tax years beginning after December 31, 2017 (increasing to 13.125 percent for tax years beginning after December 31, 2025), with a partial offset by foreign tax credits; and ?\n\nthe introduction of a territorial tax system beginning in 2018 by providing a 100 percent dividend received deduction on certain qualified dividends from foreign subsidiaries.\nDuring the fourth quarter of 2017, we recorded an income tax benefit of $1,272.4 million, which was comprised of the following: ?\nincome tax benefit of $715.0 million for the one-time deemed repatriation of foreign earnings.\n\nThis is composed of a $1,181.0 million benefit from the removal of a deferred tax liability we had recorded for the repatriation of foreign earnings prior to the 2017 Tax Act offset by $466.0 million for the toll charge recognized under the 2017 Tax Act.\nIn accordance with the 2017 Tax Act, we expect to elect to pay the toll charge in installments over eight years.\n\nAs of December 31, 2017, we have recorded current and non-current income tax liabilities related to the toll charge of $82.0 million and $384.0 million, respectively.  ?\nan income tax benefit of $557.4 million, primarily related to the remeasurement of our deferred tax assets and liabilities at the enacted corporate income tax rate of 21 percent.\n\nThe net benefit recorded was based on currently available information and interpretations made in applying the provisions of the 2017 Tax Act as of the time of filing this Annual Report on Form 10-K.  We further refined our estimates related to the impact of the 2017 Tax Act subsequent to the issuance of our earnings release for the fourth quarter of 2017.\n\nIn accordance with authoritative guidance issued by the SEC, the income tax effect for certain aspects of the 2017 Tax Act represent provisional amounts for which our accounting is incomplete, but with respect to which a reasonable estimate could be determined and recorded during the fourth quarter of 2017.\n\nThe actual effects of the 2017 Tax Act and final amounts recorded may differ materially from our current estimate of provisional amounts due to, among other things, further interpretive guidance that may be issued by U. S.  tax authorities or regulatory bodies, including the SEC and the FASB.\n\nWe will continue to analyze the 2017 Tax Act and any additional guidance that may be issued so we can finalize the full effects of applying the new legislation on our financial statements in the measurement period, which ends in the fourth quarter of 2018.\nWe continue to evaluate the impacts of the 2017 Tax Act and consider the amounts recorded to be provisional.\n\nIn addition, we are still evaluating the GILTI provisions of the 2017 Tax Act and their impact, if any, on our consolidated financial statements as of December 31, 2017.\nThe FASB allows companies to adopt an accounting policy to either recognize deferred taxes for GILTI or treat such as a tax cost in the year incurred.\n\nWe have not yet determined which accounting policy to adopt because determining the impact of the GILTI provisions requires analysis of our existing legal entity structure, the reversal of our U. S.  GAAP and U. S.  tax basis differences in the assets and liabilities of our foreign subsidiaries, and our ability to offset any tax with foreign tax credits.\nAs such, we did not record a deferred income tax\n\nIncome Taxes 2017 Tax Act: President signed U. S. tax reform legislation (\x80\x9c2017 Tax Act\x80\x9d) on December 22, 2017 considered enactment date.\n 2017 Tax Act includes broad provisions many differ from previous U. S. tax law.\n Changes in tax law accounted for in period of enactment.\n 2017 consolidated financial statements reflect immediate tax effect of 2017 Tax Act.\n 2017 Tax Act contains key provisions including ?\n\n2017 Tax Act contains key provisions including ?\n one-time tax on mandatory deemed repatriation of post-1986 untaxed foreign earnings and profits referred toll charge ?\n reduction in corporate income tax rate from 35 percent to 21 percent for tax years after December 31, 2017 ?\n\nintroduction of new U. S. tax on certain off-shore earnings as global intangible low-taxed income (GILTI) at effective tax rate of 10. 5 percent for tax years beginning after December 31, 2017 (increasing to 13. 125 percent for tax years after December 31, 2025), partial offset by foreign tax credits ?\n introduction of territorial tax system beginning in 2018 providing 100 percent dividend received deduction on certain qualified dividends from foreign subsidiaries.\n\nDuring fourth quarter of 2017 recorded income tax benefit of $1,272. 4 million of ?\n income tax benefit of $715. 0 million for one-time deemed repatriation of foreign earnings.\n composed of $1,181. 0 million benefit from removal of deferred tax liability for repatriation of foreign earnings prior to 2017 Tax Act offset by $466. 0 million for toll charge recognized under 2017 Tax Act.\n In accordance with 2017 Tax Act expect to elect to pay toll charge in installments over eight years.\n\nAs of December 31, 2017 recorded current and non-current income tax liabilities related to toll charge of $82. 0 million and $384.0 million, respectively. ?\n income tax benefit of $557. 4 million, primarily related to remeasurement of deferred tax assets and liabilities at enacted corporate income tax rate of 21 percent.\n\nnet benefit recorded based on available information and interpretations in applying provisions of 2017 Tax Act as of time of filing this Annual Report on Form 10-K. refined estimates related to impact of 2017 Tax Act subsequent to issuance of earnings release for fourth quarter of 2017.\n\nIn accordance with authoritative guidance by SEC, income tax effect for certain aspects of 2017 Tax Act represent provisional amounts for accounting is incomplete, but to reasonable estimate could be determined and recorded during fourth quarter of 2017.\n actual effects of 2017 Tax Act and final amounts recorded may differ materially from current estimate of provisional amounts due to further interpretive guidance issued by U. S. tax authorities or regulatory bodies, including SEC and FASB.\n\nwill continue to analyze 2017 Tax Act and additional guidance issued can finalize full effects of applying new legislation on financial statements in measurement period, ends in fourth quarter of 2018.\n continue to evaluate impacts of 2017 Tax Act and consider amounts recorded to be provisional.\n In, still evaluating GILTI provisions of 2017 Tax Act and impact if, on consolidated financial statements as of December 31, 2017.\n\nFASB allows companies to adopt accounting policy to recognize deferred taxes for GILTI or treat such as tax cost in year incurred.\n not yet determined which accounting policy to adopt because determining impact of GILTI provisions requires analysis of existing legal entity structure, reversal of U. S. GAAP and U. S. tax basis differences in assets and liabilities of foreign subsidiaries, and ability to offset tax with foreign tax credits.\n, we did not record deferred income tax\n\n\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: Evaluate
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.0
cosine_accuracy@3	0.0049
cosine_accuracy@5	0.0146
cosine_accuracy@10	0.0146
cosine_precision@1	0.0
cosine_precision@3	0.0024
cosine_precision@5	0.0034
cosine_precision@10	0.0017
cosine_recall@1	0.0
cosine_recall@3	0.0049
cosine_recall@5	0.0146
cosine_recall@10	0.0146
cosine_ndcg@10	0.0071
cosine_mrr@10	0.0044
cosine_map@100	0.0377
dot_accuracy@1	0.0
dot_accuracy@3	0.0
dot_accuracy@5	0.0194
dot_accuracy@10	0.0243
dot_precision@1	0.0
dot_precision@3	0.0
dot_precision@5	0.0039
dot_precision@10	0.0024
dot_recall@1	0.0
dot_recall@3	0.0
dot_recall@5	0.0194
dot_recall@10	0.0243
dot_ndcg@10	0.0098
dot_mrr@10	0.0053
dot_map@100	0.0249

Training Details

Training Dataset

Unnamed Dataset

• Size: 2,268 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 50 tokens mean: 89.83 tokens max: 229 tokens	min: 35 tokens mean: 474.02 tokens max: 512 tokens

• Samples:

  sentence_0	sentence_1
  Instruct: Given a web search query, retrieve relevant passages that answer the query. Query: Title: Text:	_id
  Instruct: Given a web search query, retrieve relevant passages that answer the query. Query: Title: Text:	_id
  Instruct: Given a web search query, retrieve relevant passages that answer the query. Query: Title: Text:	_id

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• num_train_epochs: 4
• fp16: True
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	Training Loss	Evaluate_cosine_map@100
0	0	-	0.2425
0.8818	500	0.4583	-
1.0	567	-	0.0377

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.1.1
• Transformers: 4.45.2
• PyTorch: 2.5.1+cu121
• Accelerate: 1.1.1
• Datasets: 3.1.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}