Beam alignment-assisted backscatter exhibits the enormous potential to improve the transmission rate and regulate the electromagnetic direction simultaneously. However, existing designs require decoding components with high power consumption to obtain channel state information (CSI) or provide limited beam patterns that are restricted to a specific direction. To address the above issues, we present a sector-based backscatter (SectorScatter) communication architecture that fulfills beam alignment and passive information transfer with ultralow power consumption. First, the pivotal design of SectorScatter is based on circulators and radio frequency switches, which is capable of guiding the incident wave to arbitrary sectors. Second, a two-stage beam alignment algorithm is proposed to realize adaptive directional sensing with ultralow overhead. Furthermore, a frequency-phase 16-ary modulation scheme based on limited reflection coefficients is advocated to improve the transmitting rate of passive communication. Experimental results show that SectorScatter identifies the transceiver positions in different sectors in less than $0.15~\mu $ s, communicating at a transmitting rate of up to 387 kb/s at 20 m with adaptive beam alignment toward arbitrary sectors.