Avi Ziv
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Avi Ziv has a total of 45 co-authors. Here are the top co-authors by number of shared publications:
- Raviv Gal
IBM Research
6 shared publications - Bilal Salih
IBM Research, Haifa
3 shared publications - Allon Adir
IBM Research - Israel
2 shared publications - Brian Irwin
Department of Earth, Ocean and Atmospheric Sciences, The University of British Columbia, Vancouver, BC, Canada
2 shared publications - Eldad Haber
Department of Earth, Ocean and Atmospheric Sciences The University of British Columbia Vancouver BC Canada
2 shared publications - Laurent Fournier
IBM Research Laboratory in Haifa
2 shared publications
Latest Publications
- Open Access
Automated Validation of LLM-based Evaluators for Software Engineering Artifacts
Automation in software engineering increasingly relies on large language models (LLMs) to generate, review, and assess code artifacts. However, establishing LLMs as reliable evaluators remains an open challenge: human evaluations are costly, subjective and non scalable, while existing automated methods fail to discern fine grained variations in artifact quality. We introduce REFINE (Ranking Evaluators for FIne grained Nuanced Evaluation), an automated framework for benchmarking LLM based evaluators across software engineering tasks. REFINE comprises of two modules: Hierarchy Dataset Builder applies novel generation techniques to automatically synthesize artifacts with progressively reduced quality, and Evaluator Tester quantifies each candidate evaluator configuration by measuring how closely its rankings align with expected ordering. A key feature of REFINE is controllability: users can tune the granularity of degradation to progressively refine evaluator configurations, from coarse filtering to stress testing on subtle quality gaps. While the methodology is general, we focus on coding tasks reflecting the practical demands in our production setting. REFINE was integrated into IBM's internal development workflows and applied to code generation, translation, and summarization for COBOL, an enterprise critical programming language, using industrial data. It was used to identify LLM as a Judge configurations that lifted alignment scores from below $0.7$ to above $0.9$ in some coding tasks. These nuance sensitive evaluators are now actively used by model training teams to support model release decisions.Open AccessAutomated Validation of LLM-based Evaluators for Software Engineering Artifacts
Automation in software engineering increasingly relies on large language models (LLMs) to generate, review, and assess code artifacts. However, establishing LLMs as reliable evaluators remains an open challenge: human evaluations are costly, subjective and non scalable, while existing automated methods fail to discern fine grained variations in artifact quality. We introduce REFINE (Ranking Evaluators for FIne grained Nuanced Evaluation), an automated framework for benchmarking LLM based evaluators across software engineering tasks. REFINE comprises of two modules: Hierarchy Dataset Builder applies novel generation techniques to automatically synthesize artifacts with progressively reduced quality, and Evaluator Tester quantifies each candidate evaluator configuration by measuring how closely its rankings align with expected ordering. A key feature of REFINE is controllability: users can tune the granularity of degradation to progressively refine evaluator configurations, from coarse filtering to stress testing on subtle quality gaps. While the methodology is general, we focus on coding tasks reflecting the practical demands in our production setting. REFINE was integrated into IBM's internal development workflows and applied to code generation, translation, and summarization for COBOL, an enterprise critical programming language, using industrial data. It was used to identify LLM as a Judge configurations that lifted alignment scores from below $0.7$ to above $0.9$ in some coding tasks. These nuance sensitive evaluators are now actively used by model training teams to support model release decisions. - Open Access
Quality Evaluation of COBOL to Java Code Transformation
We present an automated evaluation system for assessing COBOL-to-Java code translation within IBM's watsonx Code Assistant for Z (WCA4Z). The system addresses key challenges in evaluating LLM-based translators, including model opacity and the complexity of translation quality assessment. Our approach combines analytic checkers with LLM-as-a-judge (LaaJ) techniques to deliver scalable, multi-faceted evaluations. The system supports continuous integration workflows, enables large-scale benchmarking, and reduces reliance on manual review. We describe the system architecture, evaluation strategies, and reporting mechanisms that provide actionable insights for developers and project managers, facilitating the evolution of high-quality, modernized codebases.
Open AccessQuality Evaluation of COBOL to Java Code Transformation
We present an automated evaluation system for assessing COBOL-to-Java code translation within IBM's watsonx Code Assistant for Z (WCA4Z). The system addresses key challenges in evaluating LLM-based translators, including model opacity and the complexity of translation quality assessment. Our approach combines analytic checkers with LLM-as-a-judge (LaaJ) techniques to deliver scalable, multi-faceted evaluations. The system supports continuous integration workflows, enables large-scale benchmarking, and reduces reliance on manual review. We describe the system architecture, evaluation strategies, and reporting mechanisms that provide actionable insights for developers and project managers, facilitating the evolution of high-quality, modernized codebases.
- Open Access
- 16 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 196-197
Hybrid Cloud Connector: Offloading integration complexities
Regulated enterprises often seek to extend their workloads into the cloud, but are impeded by integration concerns relating to security, governance and compliance. Further, enterprises running mission-critical applications, face throughput and latency challenges due to cloud integration overheads. We present Hybrid Cloud Connector to accelerate on-prem to cloud integration by handling non-functional aspects in lieu of the application, reducing complexity, and centralizing administration via a policy-driven control point.
- 16 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 196-197
Open AccessHybrid Cloud Connector: Offloading integration complexities
Regulated enterprises often seek to extend their workloads into the cloud, but are impeded by integration concerns relating to security, governance and compliance. Further, enterprises running mission-critical applications, face throughput and latency challenges due to cloud integration overheads. We present Hybrid Cloud Connector to accelerate on-prem to cloud integration by handling non-functional aspects in lieu of the application, reducing complexity, and centralizing administration via a policy-driven control point.
- Open Access
- 31 March 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 28 (2) , 1-2
Introduction to the Special Issue on Machine Learning for CAD/EDA
- 31 March 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 28 (2) , 1-2
Open AccessIntroduction to the Special Issue on Machine Learning for CAD/EDA
- 6 September 2022
- Published by Springer Nature
- p. 175-225
Data Analytics and Machine Learning for Coverage Closure
- 6 September 2022
- Published by Springer Nature
- p. 175-225
Data Analytics and Machine Learning for Coverage Closure
- 10 August 2022
- Published by Springer Nature
- p. 377-424
Machine Learning in the Service of Hardware Functional Verification
- 10 August 2022
- Published by Springer Nature
- p. 377-424
Machine Learning in the Service of Hardware Functional Verification
- 26 May 2020
- Published by Springer Nature in Optimization and Engineering
- Vol. 22 (1) , 217-245
How to catch a lion in the desert: on the solution of the coverage directed generation (CDG) problem
- 26 May 2020
- Published by Springer Nature in Optimization and Engineering
- Vol. 22 (1) , 217-245
How to catch a lion in the desert: on the solution of the coverage directed generation (CDG) problem
- Open Access
How To Catch A Lion In The Desert -- On The Solution Of The Coverage Directed Generation (CDG) Problem
The testing and verification of a complex hardware or software system, such as modern integrated circuits (ICs) found in everything from smartphones to servers, can be a difficult process. One of the most difficult and time-consuming tasks a verification team faces is reaching coverage closure, or hitting all events in the coverage space. Coverage-directed-generation (CDG), or the automatic generation of tests that can hit hard-to-hit coverage events, and thus provide coverage closure, holds the potential to save verification teams significant simulation resources and time. In this paper, we propose a new approach to the CDG problem by formulating the CDG problem as a noisy derivative free optimization (DFO) problem. However, this formulation is complicated by the fact that derivatives of the objective function are unavailable, and the objective function evaluations are corrupted by noise. We solve this noisy optimization problem by utilizing techniques from direct optimization coupled with a robust noise estimator, and by leveraging techniques from inverse problems to estimate the gradient of the noisy objective function. We demonstrate the efficiency and reliability of this new approach through numerical experiments with an abstract model of part of IBM's NorthStar processor, a superscalar in-order processor designed for servers.
Open AccessHow To Catch A Lion In The Desert -- On The Solution Of The Coverage Directed Generation (CDG) Problem
The testing and verification of a complex hardware or software system, such as modern integrated circuits (ICs) found in everything from smartphones to servers, can be a difficult process. One of the most difficult and time-consuming tasks a verification team faces is reaching coverage closure, or hitting all events in the coverage space. Coverage-directed-generation (CDG), or the automatic generation of tests that can hit hard-to-hit coverage events, and thus provide coverage closure, holds the potential to save verification teams significant simulation resources and time. In this paper, we propose a new approach to the CDG problem by formulating the CDG problem as a noisy derivative free optimization (DFO) problem. However, this formulation is complicated by the fact that derivatives of the objective function are unavailable, and the objective function evaluations are corrupted by noise. We solve this noisy optimization problem by utilizing techniques from direct optimization coupled with a robust noise estimator, and by leveraging techniques from inverse problems to estimate the gradient of the noisy objective function. We demonstrate the efficiency and reliability of this new approach through numerical experiments with an abstract model of part of IBM's NorthStar processor, a superscalar in-order processor designed for servers.
- 28 November 2015
- Published by Springer Nature
- p. 51-66
The Verification Cockpit – Creating the Dream Playground for Data Analytics over the Verification Process
- 28 November 2015
- Published by Springer Nature
- p. 51-66
The Verification Cockpit – Creating the Dream Playground for Data Analytics over the Verification Process
- 12 July 2012
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 31 (8) , 1297-1302
Concurrent Generation of Concurrent Programs for Post-Silicon Validation
The continuing trend toward increased parallelism in processor design can be seen in both the growing number of processor cores per system and in on-core hardware mechanisms that assist parallelism, such as multithreading and cache hierarchies. This complexity exacerbates the problem of ensuring the functional correctness of such hardware systems. The growing importance of post-silicon validation is leading to an emerging type of parallel application, namely, the hardware exerciser. We describe a method for exercising parallel hardware by generating pseudorandom concurrent test programs. The test generation is carried out on the tested parallel platform and thus the generator itself is also a concurrent program. We describe the challenges associated with this technology and the approach used by the Threadmill hardware exerciser, a tool developed for the post-silicon validation of the IBM POWER7 processor.
- 12 July 2012
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 31 (8) , 1297-1302
Concurrent Generation of Concurrent Programs for Post-Silicon Validation
The continuing trend toward increased parallelism in processor design can be seen in both the growing number of processor cores per system and in on-core hardware mechanisms that assist parallelism, such as multithreading and cache hierarchies. This complexity exacerbates the problem of ensuring the functional correctness of such hardware systems. The growing importance of post-silicon validation is leading to an emerging type of parallel application, namely, the hardware exerciser. We describe a method for exercising parallel hardware by generating pseudorandom concurrent test programs. The test generation is carried out on the tested parallel platform and thus the generator itself is also a concurrent program. We describe the challenges associated with this technology and the approach used by the Threadmill hardware exerciser, a tool developed for the post-silicon validation of the IBM POWER7 processor.