Image-to-Audio Synthesis: Convert any image to audio by treating it as a spectrogram
Perceptual Accuracy: Uses Mel-scale frequency mapping for better perceptual results
Spectral Peak Detection: Only synthesizes significant frequency components
Temporal Smoothing: Maintains coherent trajectories between time frames
Auto-Detection: Automatically handles different image types (spectrograms vs diagrams)

Quick Start

import { synthesizeFromImage, downloadWAV } from './spectral-synthesis'

// Simple usage
const audioData = synthesizeFromImage(imageData)

// With custom parameters
const audioData = synthesizeFromImage(imageData, {
  duration: 10,
  minFreq: 100,
  maxFreq: 10000,
  maxPartials: 200,
})

// Export as WAV
downloadWAV(audioData, 44100, 'my-audio.wav')

API Reference

Main Functions

`synthesizeFromImage(imageData, params?)`

imageData: ImageData - Canvas image data
params: Partial<SynthesisParams> - Optional parameters
Returns: Float32Array - Audio samples

Types

`SynthesisParams`

interface SynthesisParams {
  duration: number // Audio duration in seconds
  minFreq: number // Minimum frequency in Hz
  maxFreq: number // Maximum frequency in Hz
  sampleRate: number // Sample rate in Hz
  frequencyResolution: number // Frequency bin downsampling
  timeResolution: number // Time slice downsampling
  amplitudeThreshold: number // Minimum amplitude threshold
  maxPartials: number // Maximum simultaneous partials
}

Project Structure

spectral-synthesis/
├── core/
│   ├── types.ts       # Type definitions
│   ├── utils.ts       # Helper functions
│   └── synthesizer.ts # Main synthesis logic
├── audio/
│   └── export.ts      # Audio export utilities
└── index.ts           # Main exports

Algorithm

Image Analysis: Auto-detect if colors should be inverted
Frequency Mapping: Convert image rows to Mel-scale frequencies
Peak Detection: Find significant spectral components
Temporal Smoothing: Apply continuity between time frames
Perceptual Weighting: Apply psychoacoustic amplitude scaling
Additive Synthesis: Generate and sum sine waves

Usage Examples

Basic Synthesis

const canvas = document.createElement('canvas')
const ctx = canvas.getContext('2d')
// ... load image to canvas
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height)
const audio = synthesizeFromImage(imageData)

Advanced Usage

import { ImageToAudioSynthesizer } from './spectral-synthesis'

const synthesizer = new ImageToAudioSynthesizer({
  duration: 5,
  maxPartials: 150,
})

const result = synthesizer.synthesize(imageData)
console.log(`Generated ${result.duration}s of audio`)