arxiv-neuroscience-research-monitor

---
name: arxiv-neuroscience-research-monitor
description: "Automated neuroscience research monitoring from arXiv. Searches for latest papers in neuroscience, brain networks, neural dynamics, spiking neural networks, and computational neuroscience. Generates skills for ai_collection and updates Obsidian wiki. Activation: neuroscience research, arxiv monitor, brain papers, neural research update."
---

# ArXiv Neuroscience Research Monitor

Automated research monitoring system for tracking latest neuroscience papers from arXiv and generating skills for the ai_collection.

## Purpose

This skill automates the process of:
1. Searching arXiv for latest neuroscience papers
2. Analyzing paper content and methodology
3. Generating structured skills for ai_collection
4. Updating Obsidian research wiki

## Search Keywords

### Primary Search Terms
- `neuroscience` - General neuroscience research
- `brain network` - Brain connectivity and network analysis
- `neural dynamics` - Neural system dynamics
- `spiking neural network` - SNN research
- `computational neuroscience` - Computational methods

### Extended Keywords (for comprehensive coverage)
- `fMRI` - Functional magnetic resonance imaging
- `EEG` - Electroencephalography
- `brain decoding` - Neural decoding from brain signals
- `neural coding` - Information encoding in neural systems
- `connectome` - Brain connectivity mapping
- `brain-computer interface` - BCI research

### Keyword Categories
| Category | Keywords |
|----------|----------|
| Imaging | fMRI, EEG, MEG, PET, calcium imaging |
| Methods | brain decoding, neural coding, connectome analysis |
| Systems | brain network, neural dynamics, spiking neural network |
| Applications | brain-computer interface, neuroprosthetics, neural rehabilitation |
| Theory | computational neuroscience, theoretical neuroscience, neural modeling |

## arXiv Categories

- `q-bio.NC` - Neurons and Cognition
- `cs.NE` - Neural and Evolutionary Computing
- `q-bio.QM` - Quantitative Methods
- `cs.LG` - Machine Learning
- `cs.AI` - Artificial Intelligence
- `cs.CV` - Computer Vision (for brain imaging)

## Workflow

### Step 0: Configure Search Parameters

```python
# Scoring weights (tuned from real execution)
SCORING_CONFIG = {
    'keyword_match': 2,           # Per keyword found in title/abstract
    'category_qbio_nc': 5,        # q-bio.NC, cs.NE, q-bio.QM
    'category_cs_lg': 2,          # Machine learning
    'category_cs_ai_cv': 1,       # AI/Computer Vision
    'recency_days': 7,            # Papers within last week
    'recency_bonus': 2            # Bonus for very recent papers
}

# Search configuration
SEARCH_CONFIG = {
    'keywords': [
        # Primary keywords
        "neuroscience",
        "brain network", 
        "neural dynamics",
        "spiking neural network",
        "computational neuroscience",
        # Extended keywords for comprehensive coverage
        "fMRI",
        "EEG",
        "brain decoding",
        "neural coding",
        "connectome",
        "brain-computer interface"
    ],
    'max_results_per_query': 20,  # Increased for extended keywords
    'days_back': 14,              # Look back 2 weeks
    'top_n_papers': 5,            # Select top N for skill generation
    'deduplication': True,        # Remove duplicate papers across keywords
    'sort_by': 'submittedDate'    # Sort by date (most recent first)
}
```

### Step 1: Search arXiv (with Fallback)

```python
import httpx
import xml.etree.ElementTree as ET
import os

ARXIV_API = "https://export.arxiv.org/api/query"  # HTTPS required (HTTP returns 301)

async def search_arxiv_neuroscience():
    """Search arXiv with fallback to local skill analysis."""
    queries = [
        "neuroscience",
        "brain network",
        "neural dynamics",
        "spiking neural network",
        "computational neuroscience"
    ]
    
    all_papers = []
    network_available = True
    
    for query in queries:
        params = {
            "search_query": f"all:{query}",
            "max_results": 20,
            "sortBy": "submittedDate",
            "sortOrder": "descending"
        }
        
        try:
            async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
                response = await client.get(ARXIV_API, params=params)
                response.raise_for_status()
            # Parse XML and extract papers...
        except Exception as e:
            print(f"Network error for '{query}': {e}")
            network_available = False
            break
    
    # Fallback: if network fails, analyze existing skills
    if not network_available or not all_papers:
        print("Network unavailable. Falling back to local skill analysis...")
        return analyze_existing_skills()
    
    return all_papers

def analyze_existing_skills():
    """Analyze existing ai_collection skills when network is down."""
    import re
    
    ai_collection_dir = os.path.expanduser("~/.hermes/skills/ai_collection")
    skill_data = []
    
    for item in os.listdir(ai_collection_dir):
        skill_path = os.path.join(ai_collection_dir, item)
        if os.path.isdir(skill_path):
            skill_md = os.path.join(skill_path, "SKILL.md")
            if os.path.exists(skill_md):
                with open(skill_md, 'r') as f:
                    content = f.read()
                
                # Extract arXiv references
                arxiv_matches = re.findall(r'arXiv[:\s]+(\d{4}\.\d+)', content)
                date_matches = re.findall(r'20\d{2}-\d{2}-\d{2}', content)
                
                skill_data.append({
                    'name': item,
                    'arxiv_ids': arxiv_matches,
                    'dates': date_matches
                })
    
    return skill_data
```

### Step 2: Filter and Rank

Selection criteria:
1. **Recency** - Published within last 30 days
2. **Relevance** - Matches research keywords
3. **Novelty** - New methodology or significant improvement
4. **Impact** - Citation potential, practical applications

### Step 3: Generate Skill

Skill structure:
```markdown
---
name: <paper-short-name>
description: "<brief description>. Activation: <keywords>."
---

# <Paper Title>

## Paper Information
- **Title:** <full title>
- **Authors:** <author list>
- **arXiv ID:** <id>
- **Date:** <publication date>
- **PDF:** <pdf url>

## Abstract
<abstract>

## Core Contributions
1. <contribution 1>
2. <contribution 2>

## Methodology
### Key Techniques
- <technique 1>
- <technique 2>

### Implementation
```python
# Code examples
```

## Applications
- <application 1>
- <application 2>

## References
- Paper: <citation>

## Activation Keywords
- <keyword 1>
- <keyword 2>
```

### Step 4: Update Obsidian

Create/update wiki notes:
- Weekly research summary
- Individual paper notes
- Cross-reference links
- Tag with research categories

## Research Categories

### Brain Networks
- Connectivity analysis
- Graph theory applications
- Network dynamics
- Topological data analysis

### EEG/BCI
- Brain-computer interfaces
- EEG signal processing
- Neural decoding
- Real-time applications

### Spiking Neural Networks
- SNN architectures
- Training algorithms
- Neuromorphic computing
- Energy efficiency

### Neural Control
- Control theory
- Dynamical systems
- Multi-agent systems
- Optimal control

### Computational Neuroscience
- Neural modeling
- Simulation frameworks
- Data analysis methods
- Foundation models

## Automation Schedule

Recommended frequency:
- **Daily** - Quick scan for new papers
- **Weekly** - Comprehensive review and skill generation
- **Monthly** - Research trend analysis

## Output Locations

### Skills
```
~/.hermes/skills/ai_collection/<skill-name>/SKILL.md
```

### Obsidian Notes
```
~/Documents/Obsidian Vault/Research/
  - Weekly-Report-<date>.md
  - Papers/<paper-name>.md
```

## Dependencies

- `httpx` - HTTP client for API requests
- `xml.etree.ElementTree` - XML parsing
- `PyYAML` - YAML frontmatter handling
- `python-frontmatter` - Frontmatter extraction

## Error Handling

Common issues and solutions discovered through trial and error:

### ArXiv API 502 Bad Gateway
- **Symptoms:** HTTP Error 502 from export.arxiv.org
- **Cause:** Server overload, maintenance, or upstream connectivity issues
- **Attempted Solutions:**
  - HTTP vs HTTPS endpoints
  - Alternative API endpoints (arxiv.org/api)
  - RSS feeds (rss.arxiv.org)
  - Different HTTP clients (httpx, urllib, curl)
- **Working Fallback:** Analyze existing ai_collection skills, document current inventory

### SSL Connection Errors (UNEXPECTED_EOF_WHILE_READING)
- **Symptoms:** SSL errors with Semantic Scholar, bioRxiv
- **Cause:** Network restrictions, certificate issues, or TLS version mismatches
- **Attempted Solutions:**
  - Custom SSL contexts with verify_mode=ssl.CERT_NONE
  - HTTP instead of HTTPS
  - Different user agents
- **Working Fallback:** Use local skill analysis when external APIs unavailable

### Rate Limiting
- **Cause:** Too many requests
- **Solution:** Add 3+ second delays between requests

### Complete Network Failure - Fallback Strategy

When all external APIs fail, pivot to:

1. **Analyze Existing Skills**
   ```python
   import os
   
   ai_collection_dir = "~/.hermes/skills/ai_collection"
   skills = []
   
   for item in os.listdir(ai_collection_dir):
       skill_path = os.path.join(ai_collection_dir, item)
       if os.path.isdir(skill_path):
           # Parse SKILL.md for paper references
           # Extract arXiv IDs, dates, categories
           # Generate research summary
   ```

2. **Generate Research Report**
   - Catalog existing skills by category
   - Identify research themes
   - Document paper references
   - Update Obsidian wiki with current state

3. **Create Monitoring Infrastructure**
   - Document the failure for future reference
   - Create skill for automated monitoring
   - Set up cron job for retry when network recovers

## Activation Keywords

- neuroscience research
- arxiv monitor
- brain papers
- neural research update
- research automation
- paper tracking

## Related Skills

- [[arxiv-search]] - General arXiv search
- [[skill-creator]] - Skill creation guide
- [[obsidian]] - Obsidian integration

## References

- ArXiv API Documentation: https://arxiv.org/help/api
- ArXiv RSS Feeds: https://arxiv.org/help/rss
- Obsidian Wiki: https://obsidian.md

## Pitfalls & Lessons Learned

### Multiple HTTP Clients Don't Help
When the server is down (502 Bad Gateway), switching between httpx, urllib, and curl doesn't help. The issue is upstream, not client-side.

### RSS Feeds Fail Together
ArXiv API and RSS feeds (rss.arxiv.org) share the same infrastructure. If one is down, the other likely is too.

### SSL Errors Are Often Environment-Specific
SSL EOF errors may indicate network restrictions or TLS version issues in the execution environment, not the target service.

### Subagent Delegation Doesn't Bypass Network Issues
Delegating to subagents for network access doesn't help if the underlying network environment has restrictions.

### Valuable Fallback: Local Analysis
When external APIs fail, analyzing existing local resources (skills, cached papers, notes) can still provide value and document current state.

## Real-World Execution Notes

### Execution 1: Network Failure (2026-04-12)
**Network Status:** All academic APIs unavailable  
**Outcome:** Successfully pivoted to local skill analysis  
**Deliverables:**
- Analyzed 44 existing ai_collection skills
- Created comprehensive research report
- Updated Obsidian wiki with current inventory
- Generated reusable monitoring skill

**Key Insight:** The fallback strategy of analyzing existing skills proved more valuable than expected, revealing research patterns and gaps that wouldn't have been visible from just fetching new papers.

### Execution 2: Successful Network Access (2026-04-12)
**Network Status:** ArXiv API accessible via HTTPS  
**Outcome:** Successfully searched, analyzed, and generated skills  
**Deliverables:**
- Searched 5 keywords, found 52 unique papers
- Scored papers using multi-factor algorithm (keywords + categories + date)
- Selected top 3 papers for deep analysis
- Generated 3 new skills for ai_collection:
  - `meta-learning-in-context-brain-decoding` (arXiv:2604.08537v1)
  - `maximum-heterogeneity-principle` (arXiv:2604.07602v1)
  - `calcium-foundation-model` (arXiv:2604.04958v2)
- Updated INDEX.md with new skill categories
- Created 3 Obsidian wiki paper notes
- Updated weekly research report

**Key Technical Lessons:**
1. **HTTPS Required**: ArXiv API now requires HTTPS (returns 301 redirect for HTTP)
2. **Follow Redirects**: Must set `follow_redirects=True` in httpx client
3. **Scoring Algorithm**: Multi-factor scoring (keywords=2pts, q-bio.NC/cs.NE=5pts, cs.LG=2pts, recent date=2pts) effectively ranks papers
4. **Deduplication**: Essential when searching multiple keywords - 75 raw results → 52 unique papers

**Scoring Formula Used:**
```python
def score_paper(paper):
    score = 0
    text = (paper['title'] + ' ' + paper['abstract']).lower()
    
    # Keywords: +2 each
    for kw in ['brain', 'neural', 'neuroscience', ...]:
        if kw in text:
            score += 2
    
    # Categories
    for cat in paper['categories']:
        if cat in ['q-bio.NC', 'cs.NE', 'q-bio.QM']:
            score += 5  # High value
        elif cat == 'cs.LG':
            score += 2
        elif cat in ['cs.AI', 'cs.CV']:
            score += 1
    
    # Recency
    if paper['published'] >= '2026-04-08':
        score += 2
    
    return score
```

**Skill Generation Pattern:**
1. Extract core methodology from abstract
2. Identify activation keywords (English + Chinese)
3. Define use cases and applications
4. Create structured SKILL.md with frontmatter
5. Update ai_collection INDEX.md
6. Create Obsidian paper notes
7. Update weekly research report

**Skill Naming Conventions:**
- Use lowercase with hyphens: `paper-name-methodology`
- For updated versions of existing skills, use `-v2`, `-v3` suffix: `meta-learning-brain-decoding-v2`
- Keep names concise but descriptive (max 3-4 words)
- Include methodology or key technique in name

---

*Skill Type: Automation Workflow*
*Category: Research*
*Created: 2026-04-12*
*Last Updated: 2026-04-12 (added extended keywords, deduplication config, naming conventions)*