Repository navigation
How to create code2vec input #186
Description
I use command like “{java -cp JavaExtractor-0.0.1-SNAPSHOT.jar JavaExtractor.App --max_path_length 8 --max_path_width 2 --dir test.java >file.txt }“ ，then use ”{python3 code2vec.py --load models/java14_model/saved_model_iter8.release --test file.txt}“，but get error “ {return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Expect 201 fields but have 4 in record
[[{{node IteratorGetNext}}]] }”.
Activity
urialon commentedon Nov 16, 2023
Hi @messiGao ,
Thank you for your interest in our work.
I think there is a confusion, because the exception that is raised is coming from TensorFlow, while the java command that you mentioned does not involve TensorFlow at all.
May I also ask what kinds of tasks are you looking into?
Maybe I can recommend a newer model.
Best,
Uri
messiGao commentedon Nov 16, 2023
I want to use the “--test” command to export <TEST_FILE>.vectors,but I don't know what kind of TEST_FILE is correct。when i ask gpt-4， the answer is use the JavaExtractor to convert my test.java to test.txt。
messiGao commentedon Nov 16, 2023
Additionally,My aim is to store a Java codebase in a vector database to run similarity searches and retrieve code files from the db relevant to my query.
urialon commentedon Nov 16, 2023
Hi @messiGao ,
Please see https://github.com/neulab/code-bert-score
You don't need the approach itself, but it contains Huggingface models, and one specifically for java called
neulab/codebert-java.This will allow you to use the Huggingface library with that model and a BERT-like framework.
Best,
Uri
asyed79gatech commentedon Feb 22, 2024
I have a similar dilemma with regards to creating embeddings of csharp code using a code2vec model I have trained. As
@messiGao mentioned, I want to use the "--test" command to create .vectors file as mentioned in the repo but when i execute the command, it gives the following error:
urialon commentedon Feb 22, 2024
Hi @asyed79gatech ,
Thank you for your interest in our work.
I believe that you haven't run the
preprocess.shscript on the data.However in general, I recommend using the newer https://github.com/neulab/code-bert-score project. It is based on Huggingface, which is actively maintained.
Best,
Uri
asyed79gatech commentedon Feb 22, 2024
Hi @urialon
Thanks for your prompt response. I thought we only needed to run the preprocess.sh script while training the code2vec model. Right now, I already have a trained model released and want it to generate embeddings for vector store.
XuPing1234 commentedon Jun 29, 2024
Hello, have you resolved your issue? How can Java source code be converted into the input format required by code2vec?
zhaojialinnn commentedon Aug 20, 2024
hello, I encountered the same issue. Have you resolved it?