MSRA Proposal Review Skill

⚠️ CRITICAL: ACCURACY POLICY AND AVOID HALLUCINATION

ALL content in the review MUST be grounded in the actual proposal text.

• •✅ EXTRACT all relevant information that IS in the proposal
• •✅ Read the ENTIRE PDF output carefully
• •❌ NEVER invent information that is NOT in the document
• •❌ When information is genuinely absent, use "N/A"

Purpose

Review and rate a single MSRA research collaboration proposal using pre-generated rubrics. This skill produces a structured assessment following the assessment template.

When to Activate

Activate ONLY when the user explicitly asks to:

• •Review a specific MSRA proposal (with proposal name provided)
• •Rate a single StarTrack application
• •Evaluate a specific funding proposal for Jianxun Lian's collaboration

Prerequisites:

• •Rubrics must already be generated (using msra-rubric-generation skill)
• •A specific proposal name must be provided

Input Requirements

The user must provide:

• •Input folder: Directory containing proposal PDFs
• •Proposal Name: Name of the specific proposal/applicant to review
• •Output folder: Directory to save the assessment
• •Rubrics folder (optional): If different from output folder, where rubrics.json is located

Output

• •<ApplicantName>_assessment.md - The structured assessment following the template

Reference Files

Before starting, read these reference files for context:

1. •

   references/jianxun_profile.md - Jianxun Lian's research background, focus areas, and ideal collaborator profile. ESSENTIAL for evaluating the alignment dimension.

2. •

   references/assessment_template.md - The exact template format for generating assessment reports. ALL assessments must follow this structure.

Workflow

Step 1: Load Prerequisites

1. •

   Load rubrics:

   code

   run_skill_script("load_rubrics", ["<rubrics_folder>"])
   

   • •If rubrics not found, inform user to run rubric generation first
   • •Confirm rubrics are frozen/stable
2. •

   Find the proposal PDF:

   code

   run_skill_script("find_proposal", ["<input_folder>", "<proposal_name>"])
   

Step 2: Read and Analyze Proposal

1. •

   Read the proposal PDF:

   code

   run_skill_script("read_proposal", ["<pdf_path>"])
   

2. •

   Extract key information (scan the ENTIRE document including later pages):

   • •Full name of applicant
   • •Organization/University
   • •QS World University Ranking (2026): This is NOT in the proposal. You MUST use web_search tool to look up the QS ranking for the applicant's university. Search for "[University Name] QS ranking 2026" or check the official QS World University Rankings.
   • •Senior Professional Service: Extract roles such as General Chair or PC Chair of top-tier conferences (e.g., KDD, ACL, NeurIPS, ICML, AAAI, SIGIR, CHI, WWW, CVPR, ICLR). Exclude workshop chairs, PC members, area chairs, or reviewers. List all qualifying roles (e.g., "General Chair of KDD 2024, PC Chair of SIGIR 2023"). If none found, use "N/A".
   • •Awards: Extract significant awards from the proposal such as Best Paper Awards, Best Paper Honorable Mentions, Test of Time Awards, Rising Stars, NSF CAREER Award, ACM/IEEE Fellow, Microsoft Research Fellowship, or similar prestigious recognitions. Select top 5 most significant if there are over 5 awards. Rank the order by the award significance. Use "N/A" only if no awards are mentioned.
   • •Country
   • •Whether recommendation letter is included: Carefully check ALL pages, especially the last few pages. Look for:
     • •"To whom it may concern"
     • •"I write in support of..."
     • •"Letter of Support"
     • •"Recommendation Letter"
     • •Text written by someone OTHER than the applicant endorsing the proposal
     • •If ANY such letter exists ANYWHERE in the PDF, set has_letter: true
   • •Top 5 representative papers

Step 3: Apply Rubrics and Score

Score the proposal on exactly 6 dimensions (1-10 scale):

For each dimension:

1. •Consult the rubrics anchors for that dimension
2. •Compare against anchor descriptions
3. •Note applicable weakness/strength patterns
4. •Assign score with justification

Step 4: Generate Assessment

Use the save_assessment script to generate the final output:

run_skill_script("save_assessment", ["<applicant_name>", "<output_folder>", "<assessment_json>"])

REQUIRED assessment_json structure:

{
  "basic_info": {
    "name": "Full Name",
    "organization": "University Name",
    "qs_ranking": 50,
    "senior_service": "General Chair of KDD 2024, PC Chair of SIGIR 2023",
    "awards": "Best Paper Award at CHI 2022, NSF CAREER Award 2021",
    "country": "Country",
    "has_letter": false
  },
  "top_papers": [
    "Paper 1, Venue Year",
    "Paper 2, Venue Year",
    "Paper 3, Venue Year",
    "Paper 4, Venue Year",
    "Paper 5, Venue Year"
  ],
  "scores": {
    "excitement": 5,
    "impact": 4,
    "feasibility": 6,
    "completeness": 5,
    "collaboration": 4,
    "alignment": 5
  },
  "patterns": ["W003", "S002"],
  "justifications": {
    "excitement": "Detailed justification referencing rubric anchors...",
    "impact": "Detailed justification...",
    "feasibility": "Detailed justification...",
    "completeness": "Detailed justification...",
    "collaboration": "Detailed justification...",
    "alignment": "Detailed justification..."
  },
  "final_summary": "MUST include **Weaknesses:** section (2-4 sentences) and **Strengths:** section (1-2 sentences). Weaknesses MUST be longer than strengths. See format rules below."
}

final_summary Format Rules

The final_summary will be pasted to a form WITHOUT rubrics. Follow these rules strictly:

1. •DO NOT include pattern IDs like (S001), (W003) - describe strengths/weaknesses in plain language
2. •DO NOT include decision recommendations like "recommended for consideration", "should be prioritized"
3. •DO NOT mention Jianxun Lian by name
4. •⚠️ WEAKNESSES ARE MORE IMPORTANT THAN STRENGTHS - see structure below
5. •Keep it factual and neutral in tone

⚠️ CRITICAL: final_summary Structure (MUST FOLLOW)

Weaknesses are MORE important than strengths for decision-making. The summary MUST follow this structure:

[1 sentence: What the proposal aims to do]

**Weaknesses:** [2-4 sentences describing key weaknesses, limitations, concerns, or gaps. Be specific and critical. This section should be LONGER than the strengths section. NO PATTERN IDs like W001, S002.]

**Strengths:** [1-2 sentences highlighting notable strengths, if any. NO PATTERN IDs.]

⚠️ NEVER include pattern IDs (W001, S002, etc.) in final_summary - describe weaknesses/strengths in plain language only!

Example of GOOD final_summary:

The proposal aims to develop socially intelligent tutoring agents using scaffolded hints and agentic memory for long-term learning.

Weaknesses: The technical approach is largely incremental, applying existing LLM techniques (DPO, SFT, CoT) without clear novelty in the core methodology. The proposal lacks detailed risk mitigation and contingency plans. The connection to Theory-of-Mind and multi-agent coordination—central themes of the research track—is weak, framing ToM as a secondary consideration rather than a core technical contribution. Collaboration plans mention Microsoft generically without naming specific researchers or prior ties.

Strengths: The PI has a strong publication record in hint generation and QA datasets. The proposal is well-structured with clear deliverables.

Example of BAD final_summary (DO NOT DO THIS):

This proposal is strong (S001, S002). The PI has good experience (S008). The work is well-aligned and feasible. Recommended for high priority collaboration support.

Scoring Guidelines

⚠️ CRITICAL: Be Selective

Most proposals should score 4-5 (Below Baseline to Baseline).

Use the rubrics anchors to calibrate your scores:

Collaboration Scoring (UPDATED)

The collaboration dimension values EXISTING ties highly:

Key insight: A 2025 MSR internship or prior joint work is STRONGER evidence than a detailed future plan alone. Score 7+ for existing ties.

"Natural fit for joint work" = Score 4-5 (no explicit willingness shown)

Using Rubric Patterns

Always reference pattern IDs from the loaded rubrics.json in justifications:

• •Example: "Low excitement due to W002 (Low Significance) — the problem is already well-studied"
• •Example: "Strong collaboration evidence (S008: Proven Microsoft Collaboration) — had MSR internship in 2025"

Important: The specific patterns available depend on the rubrics generated. Always check the loaded rubrics for:

• •weakness_patterns: List of weakness IDs (W001, W002, ...) with descriptions
• •strength_patterns: List of strength IDs (S001, S002, ...) with descriptions

Use the pattern IDs and names exactly as they appear in the rubrics.

Recommendation Mapping

Based on total score (out of 60):

Example User Instruction

MSRA proposal review for Jianxun Lian's collaboration.

Input folder: /home/jialia/projects/tmp/agent-memory/data/research_fellowship
Proposal Name: John Smith
Output folder: /home/jialia/projects/tmp/agent-memory/output/msra-reviews